The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report changed, do nothing useful, and leave you believing the server is hardened when it is not.

This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.

What a one-run bootstrap actually has to survive

A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.

Four things break it, and they break it in different ways:

• The chicken-and-egg problem. Ansible needs Python on the target to run almost any module, and a minimal image may not have it.
• Lockout. You remove your own access before proving the replacement access works.
• Non-idempotent tasks. The first run is fine. The second run is where the damage happens.
• Missing collections. The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with ansible-core.

Everything below is organised around those four, because that is how they actually show up.

The chicken-and-egg problem: no Python, no modules

Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.

The escape hatch is ansible.builtin.raw, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with gather_facts: false, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.

- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update && apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:

Two details worth understanding rather than copying. failed_when: false on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And ansible.builtin.setup is the fact-gathering module called explicitly, which means from that point on ansible_distribution and friends are available even though the play started with gathering disabled.

If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.

Failure family one: locking yourself out while the playbook says ok

This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.

The order that survives

The rule is simple: nothing that removes access happens until something has proven the new access works. In practice:

1. Connect as whatever the provider handed you, usually root or a sudo-capable default user.
2. Create the admin account, add it to the sudo or wheel group, install its authorized key.
3. Write the sudoers drop-in and validate it with visudo -cf.
4. Prove you can log in as that account and escalate. This is the step everyone skips.
5. Install the firewall and allow SSH before enabling it.
6. Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.
7. Only now disable root login and password authentication.

Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:

- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"

Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.

The socket activation trap

This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. ssh.socket owns the listening port, and Port in sshd_config is not what decides where the daemon listens.

So your playbook writes Port 2222, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.

The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:

# Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'

If ssh.socket is enabled, the override needs an empty ListenStream= first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.

Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.

Firewall ordering

UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an UNREACHABLE that will not recover. Rules go in first, always:

- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled

Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.

Failure family two: the second run is the dangerous one

A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are shell and command tasks with no guard.

An unguarded shell task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.

Guard them properly:

• creates: or removes: makes the task skip when the target state already exists. Prefer this.
• changed_when: false for read-only commands so they stop lying about changing things.
• when: driven by a registered check, for anything conditional.

The other idempotency trap is lineinfile against sshd_config. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:

- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false

Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so validate: on the template task with sshd -t -f %s will fail on a temporary file that has no host keys and no context. Run sshd -t as its own task afterwards instead, and let the handler restart only if it passes.

Two prerequisites for the drop-in approach. The base sshd_config needs an Include /etc/ssh/sshd_config.d/*.conf line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the first value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.

Failure family three: the collections are not where you think

ansible-core ships the ansible.builtin namespace and essentially nothing else. The full ansible package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.

Things people assume are built in and are not: ufw and timezone live in community.general. authorized_key, sysctl and mount live in ansible.posix. Declare them and pin them:

# requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml

Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. ansible-lint flags this and it is worth listening to.

Failure family four: secrets in the repository

Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.

Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:

ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass

The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.

Putting it together: how to bootstrap a VPS with Ansible end to end

Structure matters less than order, but a shape that works well is a single site.yml calling roles in dependency order, with the risky access changes fenced off at the end.

# inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22

Then the run itself, in this order:

1. Reachability. Python check, fact gathering, timezone, hostname, package cache refresh.
2. Identity. Admin user, group membership, authorized key, sudoers drop-in with visudo -cf validation.
3. Verification gate. The task shown earlier. Nothing destructive runs before this passes.
4. Firewall. Rules first, policy second, enable third.
5. Unattended updates. unattended-upgrades on Debian and Ubuntu, dnf-automatic with its systemd timer on RHEL-family hosts.
6. Brute-force protection. fail2ban with your own /etc/fail2ban/jail.local, never edits to jail.conf.
7. Workload. Docker, a web server, a monitoring agent, whatever the box is for.
8. Access teardown. Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.

When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:

- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60

meta: reset_connection does not accept a when conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.

Test it before you point it at anything real

The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.

# catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff

One caveat on --check: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.

Troubleshooting

• UNREACHABLE on the very first task. Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with ssh-keygen -R and re-add the correct one with ssh-keyscan. Disabling host key checking globally makes this go away and removes a real protection, so do not.
• UNREACHABLE immediately after a firewall task. You armed the firewall before allowing SSH. Recover through the provider console, not SSH.
• “Missing sudo password”. The sudoers drop-in is not in effect or has bad syntax. Validate with visudo -cf on the file itself before trusting it.
• Module not found under an FQCN. The collection is not installed on the machine running the playbook. Confirm with ansible-galaxy collection list.
• New group membership ignored. Adding a user to docker mid-play does not affect the already-open connection. Insert meta: reset_connection after the group change.
• fail2ban installed but not banning. On images without rsyslog, /var/log/auth.log may not exist, so a jail configured to read it never starts. Check fail2ban-client status and switch the jail to the systemd backend if the log file is absent.
• Everything green, port unchanged. Socket activation. Check systemctl is-enabled ssh.socket and ss -tlnp.

Common mistakes

• Disabling root login in the same play that creates the replacement user, with no verification step between them.
• Enabling UFW before the SSH allow rule exists.
• Assuming Port in sshd_config is authoritative on every distribution.
• Unguarded shell tasks that notify handlers, causing service restarts on every run.
• Editing jail.conf instead of creating jail.local, so the next package update silently reverts it.
• Committing plaintext secrets because “it is only a bootstrap playbook”.
• Testing on a host you cannot afford to lose, with no console access configured.
• Relying on --check as a substitute for a real run on a disposable host.

Best practices

• Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.
• Put a verification task before every irreversible one, and let the play fail there rather than after.
• Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.
• Validate configuration before restarting the service that reads it. sshd -t, visudo -cf, nginx -t.
• Pin collections in requirements.yml and install them in CI, not just locally.
• Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.
• Reboot once during testing. Half of the silent failures only appear after a restart.
• Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.

Frequently asked questions

Can you really bootstrap a VPS with Ansible in a single playbook run?

Yes, including changing the SSH user and port mid-run. The mechanism is set_fact to update the connection variables, meta: reset_connection to drop the cached connection, and wait_for_connection to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.

Should I use cloud-init or Ansible for initial server setup?

Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.

Do I need Terraform or OpenTofu as well?

Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.

Why does my playbook work the first time and fail the second?

Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.

Is changing the SSH port worth doing?

It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.

How do I recover if the playbook locks me out?

Through the provider’s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.

Can this run against several servers at once?

Yes, but use serial: 1 the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.

The one thing worth remembering

When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.

Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.

Need a bootstrap playbook you can actually rerun?

Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:

• Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.
• Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.
• Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.
• Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.
• Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.
• Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.

Send me the playbook, a recap from a second run, or the output of ss -tlnp from the host that is behaving oddly, and I will tell you what I would change first.

The post Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.

The line you wrote was real. The file was saved. What you did not know is that your distro puts an Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, that sshd uses the first value it finds for any given keyword, and that your cloud provider’s image dropped a 50-cloud-init.conf in that directory setting PasswordAuthentication yes. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.

You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.

That is the thing most guides get wrong, and it is the idea this VPS hardening checklist is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.

This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.

Before you touch anything

Two minutes of preparation that saves an evening.

Find your provider’s console before you need it. Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.

Never close the session you are working in. Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.

Then update the system, because everything else assumes current packages:

sudo apt update && sudo apt upgrade -y
sudo reboot   # if a kernel update landed

1. A real user, and keys instead of passwords

Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.

# On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"

# On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10

Now open a second terminal and confirm ssh deploy@203.0.113.10 works, and that sudo -v works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.

2. SSH config, and why your edit might do nothing

This is the section that actually matters. Two mechanisms silently override what you write.

Drop-in files and first-match-wins

Modern OpenSSH packages ship an Include line near the top of /etc/ssh/sshd_config, which pulls in every .conf file from /etc/ssh/sshd_config.d/ in alphabetical order, before the rest of the main file is read. Combine that with sshd’s rule that the first value found for a keyword wins, and the consequence is specific: your drop-in has to sort earlier than the vendor’s, not later.

Most guides tell you to name it 99-something.conf, which is the convention almost everywhere else in Linux and exactly backwards here.

# /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy

A note on KbdInteractiveAuthentication: older guides and older OpenSSH call this ChallengeResponseAuthentication. Setting only PasswordAuthentication no can still leave a keyboard-interactive path open on some configurations, which is why both are here.

Now the part nobody does. Validate the syntax, then read the effective config:

# Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'

If passwordauthentication yes comes back, your file lost to something else in that directory. Look at what is in there with ls /etc/ssh/sshd_config.d/ and either rename yours to sort earlier or edit the offending file directly.

Socket activation, and the port that will not change

The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: ssh.socket holds the listening port and starts sshd only when a connection arrives. On those systems the Port and ListenAddress directives in sshd_config may be ignored entirely, because systemd owns the socket.

People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:

# -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp

If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under /lib/systemd/system/, which a package update will overwrite:

sudo systemctl edit ssh.socket

# In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it

Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.

3. Firewall, in the right order

The ordering here is the whole trick. Set the default policy and add your rules before you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose

Read that status verbose output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.

One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with -p 5432:5432 is reachable from the internet even though ufw says that port is denied. Bind container ports to 127.0.0.1 explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.

A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.

4. Automatic security updates

Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug

By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.

5. fail2ban, and what it is actually for

Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.

What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials do exist: a mail server, a web application login form, an FTP daemon.

sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd

Check that status output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, journalctl -u fail2ban will tell you why.

Set ignoreip in jail.local to include your office or home address. Banning yourself is a rite of passage but it is avoidable.

6. The boring items that matter more than the exciting ones

Nobody writes blog posts about these, and they cause more real outages than any of the above.

Provider account security

Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.

Time and timezone

sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"

UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.

Swap on small instances

# A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

Backups that leave the machine

Your provider’s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.

Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man’s switch service at the backup job so you find out when it stops running, rather than the day you need it.

Knowing the box is alive

At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.

What I would skip

Not everything in the long hardening guides earns its cost on a single web server.

• Large sysctl hardening blocks pasted from a gist. Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.
• AIDE, auditd and file-integrity monitoring on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.
• Rebuilding OpenSSH’s cipher list. Current defaults are good. Guides that hand you a Ciphers line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.
• Disabling ping. It breaks your own monitoring and hides nothing.

Troubleshooting

Locked out after an SSH change

Provider console, log in as root, and revert. If you cannot remember what you changed, ls -lt /etc/ssh/sshd_config.d/ shows the most recently modified file. Run sshd -t to see whether the config is even valid, since a syntax error stops sshd starting at all.

Key auth rejected, password prompt appears instead

Almost always permissions. The home directory must not be group-writable, ~/.ssh needs 700 and authorized_keys 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run ssh -v to see how far it gets, and check journalctl -u ssh on the server for the actual reason.

The setting I applied is not in effect

sudo sshd -T and search for the keyword. If the effective value differs from your file, something in sshd_config.d/ sorts earlier, or a Match block later in the config is overriding it for your connection.

A service is reachable that the firewall says is blocked

Docker, or another daemon writing netfilter rules directly. Check with sudo iptables -S or sudo nft list ruleset, and always verify from a machine outside the server rather than trusting ufw status.

Nothing obviously wrong, but who has been logging in?

last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"

Common mistakes

• Closing your only SSH session before testing the new configuration from a second one.
• Enabling the firewall before adding the SSH rule.
• Naming your sshd drop-in 99-, so a vendor file that sorts earlier wins.
• Trusting the file you edited instead of running sshd -T.
• Generating the SSH key on the server.
• Changing the SSH port and counting it as a security control.
• Disabling password auth for SSH while leaving a web control panel with a weak password exposed.
• No two-factor authentication on the hosting account that can reset root.
• Treating provider snapshots as backups.
• Installing fail2ban and never checking that it started.
• Publishing container ports without binding them to localhost, then assuming ufw covers it.
• Pasting a sysctl block you cannot explain line by line.

Best practices

• Verify every control after applying it, with a command that reads the running state.
• Keep a second session open and the provider console tested before making SSH changes.
• Key-only authentication, root login disabled, and an explicit AllowUsers list.
• Default-deny inbound firewall, with rules added before the firewall is enabled.
• Automatic security updates on, with a deliberate decision about automatic reboots.
• Two-factor authentication on the hosting account and the registrar.
• Backups off the machine, on separate credentials, and restore-tested at least once.
• UTC everywhere, with clock sync confirmed.
• Monitoring that pages you on disk usage before it pages you on downtime.
• Script the whole build, so the next server is identical and you are not repeating this from memory.

FAQ

What is the single most important step?

Key-only SSH with root login disabled, verified with sshd -T. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.

Should I change the SSH port?

For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.

Do I need fail2ban if password authentication is off?

Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.

Why did my sshd_config change get ignored?

Two likely causes. A file in /etc/ssh/sshd_config.d/ that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is Port or ListenAddress on a socket-activated system, where systemd owns the listener. sshd -T answers the first case and ss -tlnp answers the second.

Is ufw enough, or should I use nftables directly?

ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.

How do I harden a server I did not build?

Audit before you change anything. sshd -T, ufw status verbose, ss -tlnp for unexpected listeners, systemctl list-units --type=service --state=running, and the contents of every authorized_keys on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.

Should this be automated?

Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.

The one thing to remember

Editing a config file is not the same as changing a system’s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.

So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. sshd -T for SSH, ufw status verbose for the firewall, ss -tlnp for what is listening, systemctl status for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.

Want a second pair of eyes on your server?

Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:

• Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.
• Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.
• Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.
• Moving SSH off the public internet onto a private overlay network or bastion.
• Backup strategy that survives the hosting account being compromised, including a tested restore.
• Firewall and container networking, especially where published Docker ports are bypassing the firewall.

Send me the output of sshd -T, ufw status verbose and ss -tlnp, and I will tell you what stands out.

The post You Probably Didn’t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself appeared first on John Nessime.

For a single server, neither is the right answer. Prometheus and Grafana are two static Go binaries and a package. They run happily under systemd, use very little memory at this scale, and when something breaks you can read the logs and actually fix it. No container networking to debug, no orchestration layer between you and the problem.

This walks through the whole thing: install Prometheus and Grafana from scratch, wire in node_exporter, put the lot behind nginx with TLS, and lock down the ports that should never have been public in the first place. Commands work on both Debian-family and RHEL-family systems, with the differences called out.

What you are building

Four moving parts, all on one box:

• node_exporter exposes machine metrics — CPU, memory, disk, network — on port 9100.
• Prometheus scrapes that endpoint on a schedule and stores the samples in its own time series database on port 9090.
• Grafana queries Prometheus and draws the dashboards on port 3000.
• nginx terminates TLS and is the only thing listening on a public interface.

The critical design decision is in that last line. Prometheus and node_exporter ship with no authentication at all. Bound to a public interface, they hand your entire infrastructure inventory to anyone who runs a port scan. Everything binds to 127.0.0.1 and nginx is the single front door.

Resource-wise, a 2 GB VPS handles this comfortably for one server’s worth of metrics. Prometheus is the memory-hungry one, and its appetite scales with the number of active time series rather than the number of hosts.

Step 1: Users and directories

Neither service should run as root, and neither needs a login shell.

sudo useradd --system --no-create-home --shell /sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /sbin/nologin node_exporter

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

On Debian and Ubuntu the nologin shell lives at /usr/sbin/nologin rather than /sbin/nologin. Check with which nologin if useradd complains.

Step 2: Install Prometheus

Prometheus ships as a single static binary. Rather than hardcoding a version that will be stale next month, pull the current release tag from the API:

PROM_VERSION=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')
echo "Installing Prometheus ${PROM_VERSION}"

cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"

sudo cp prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

If you are on ARM, swap linux-amd64 for linux-arm64.

The configuration file

Write /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
        labels:
          instance: "web-01"

Set that instance label deliberately. It follows every metric into every dashboard and alert, and renaming it later means your historical graphs split in two.

Validate before you start anything:

promtool check config /etc/prometheus/prometheus.yml

The systemd unit

Write /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --storage.tsdb.path=/var/lib/prometheus 
  --storage.tsdb.retention.time=30d 
  --storage.tsdb.retention.size=8GB 
  --web.listen-address=127.0.0.1:9090
ExecReload=/bin/kill -HUP $MAINPID

NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/prometheus
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Two things worth noticing. --web.listen-address=127.0.0.1:9090 is what keeps Prometheus off the public internet. And setting both retention flags matters — time-based retention alone will not save you if metric volume spikes, because Prometheus honours whichever limit it hits first.

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Step 3: node_exporter

Same pattern — a static binary and a unit file.

NE_VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')

cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}/node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
tar xzf "node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
sudo cp "node_exporter-${NE_VERSION}.linux-amd64/node_exporter" /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Then /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/node_exporter 
  --web.listen-address=127.0.0.1:9100

NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

# Confirm it is actually producing metrics
curl -s http://127.0.0.1:9100/metrics | head -20

Now open http://127.0.0.1:9090/targets — via an SSH tunnel, since it is not public — and both jobs should read UP.

# From your laptop
ssh -L 9090:127.0.0.1:9090 you@your-server

Step 4: Install Grafana

Grafana is packaged properly, so use the official repository rather than a tarball. It gets you signed updates through your normal patching process.

Debian and Ubuntu:

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key 
  | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" 
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y grafana

RHEL, AlmaLinux, Rocky:

sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana

Bind it to localhost too

Edit /etc/grafana/grafana.ini:

[server]
http_addr = 127.0.0.1
http_port = 3000
domain = metrics.example.com
root_url = https://metrics.example.com/

[security]
admin_user = admin
admin_password = put-something-real-here
cookie_secure = true

[users]
allow_sign_up = false

Getting root_url right matters more than it looks. Set it wrong and Grafana generates broken links and redirects behind the proxy, which produces a confusing half-working UI rather than a clean error.

sudo systemctl enable --now grafana-server

Step 5: Put nginx and TLS in front

This is the only piece that faces the internet.

server {
    listen 443 ssl;
    server_name metrics.example.com;

    ssl_certificate     /etc/letsencrypt/live/metrics.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/metrics.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Grafana Live needs websockets, and breaks silently without this
    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host       $host;
        proxy_pass http://127.0.0.1:3000;
    }
}

Note fullchain.pem rather than cert.pem. Getting that wrong produces a site that works in your browser and fails for anything else.

Then close the metrics ports at the firewall, belt and braces:

# firewalld
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# ufw
sudo ufw allow 443/tcp

# Verify nothing unexpected is listening publicly
sudo ss -tlnp | grep -E '9090|9100|3000'

Every line of that last command should show 127.0.0.1. If you see 0.0.0.0 or *, something is exposed and you should fix it before going any further.

Step 6: Connect Grafana to Prometheus

You can click through the UI, but provisioning it as a file means a rebuilt server comes back identical. Create /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true

Restart Grafana, log in, and import dashboard ID 1860 from the Grafana dashboard library — Node Exporter Full. It is comprehensive, well maintained, and will show you more about your server than you knew was measurable.

Sizing the disk before it bites

Prometheus storage is predictable. Roughly:

disk = retention_seconds × samples_per_second × bytes_per_sample

Compressed samples land in the region of one to two bytes each. Measure your actual rate rather than guessing — run this in the Prometheus query box:

rate(prometheus_tsdb_head_samples_appended_total[5m])

One server with node_exporter at a 15 second interval is a small number, and 30 days fits in a few gigabytes. The number climbs fast once you add exporters, so keep --storage.tsdb.retention.size set as a hard ceiling. Prometheus filling the root partition takes the rest of the box down with it, which is a memorable way to discover your monitoring has no monitoring.

Troubleshooting

A target shows DOWN

Work outward from the exporter:

curl -s http://127.0.0.1:9100/metrics | head
sudo systemctl status node_exporter
sudo journalctl -u prometheus -n 50 --no-pager

On RHEL-family systems with SELinux enforcing, this is frequently SELinux rather than your config. Check before you start changing things:

sudo ausearch -m avc -ts recent

If that shows denials, write a policy for them. Do not reach for setenforce 0 — it fixes today’s symptom and quietly removes a layer of protection from the whole machine.

Grafana loads but every panel says “No data”

Almost always the datasource URL. It must be http://127.0.0.1:9090 from Grafana’s point of view, not your public hostname. Test the datasource from its settings page — Grafana reports the actual connection error, which is more useful than the empty panel.

Prometheus will not start

sudo journalctl -u prometheus -n 50 --no-pager
promtool check config /etc/prometheus/prometheus.yml
ls -la /var/lib/prometheus

The usual culprits are a YAML indentation error, or /var/lib/prometheus not being owned by the prometheus user after a manual file copy.

Broken styling or redirect loops behind the proxy

root_url in grafana.ini does not match how you are actually reaching the site. If you are serving from a subpath, you also need serve_from_sub_path = true.

Common mistakes

• Leaving 9090, 9100 or 3000 on a public interface. Prometheus has no authentication whatsoever.
• Keeping the default Grafana admin password because it prompts you to change it and you clicked past.
• Setting no retention limits and discovering the problem when the root partition fills.
• Dropping scrape_interval to 1s because more data seems better. It multiplies storage and buys you nothing at this scale.
• Never backing up /var/lib/grafana/grafana.db, which holds every dashboard you have built.
• Running everything as root because the tutorial you followed did.
• Building dashboards but configuring no alerts, so you still find out from a user.

Best practices

• Bind every component to localhost and expose exactly one TLS endpoint.
• Set both time and size retention. Whichever triggers first is the one you want.
• Provision datasources and dashboards as files, so the config lives in version control.
• Back up the Grafana SQLite database on the same schedule as everything else that matters.
• Add an external uptime check. Self-hosted monitoring cannot tell you that the server it lives on has gone.
• Write a handful of alerts you will genuinely act on. Alerts nobody responds to train people to ignore alerts.
• Use systemd’s sandboxing directives. They cost nothing and limit what a compromised exporter can reach.

Frequently asked questions

Should I use Docker instead?

For a single server, systemd is simpler. Two static binaries and a package, with no container networking layer between you and the logs. Containers earn their keep once you are managing several hosts or already run everything else that way.

How much RAM does this need?

A 2 GB VPS is comfortable for one server. Prometheus memory tracks active time series rather than host count, so it grows when you add exporters, not when you add machines with the same exporters.

Can I monitor more servers later?

Yes. Install node_exporter on each and add them as targets. The moment metrics cross a network, though, that traffic needs protecting — a VPN, an SSH tunnel, or TLS with client certificates. Do not simply open 9100 to the world.

Where do alerts come from?

Two options. Grafana’s built-in alerting works from dashboard queries and is the quicker path. Alertmanager, run alongside Prometheus, is more powerful for routing, grouping and silencing. Start with Grafana’s and move if you outgrow it.

How long should I keep metrics?

Thirty days covers most troubleshooting and capacity questions. Longer retention on a single box is usually a sign you want a remote-write backend rather than a bigger disk.

Do I need node_exporter if I only care about my application?

Install it anyway. When the application misbehaves, the first question is whether the machine underneath it is healthy, and node_exporter is what answers that.

Wrapping up

Once you install Prometheus and Grafana this way, the whole stack is four systemd units and three config files. Everything binds to localhost, nginx handles TLS, and there is no layer of abstraction between you and a problem when something breaks at an inconvenient hour.

Get the dashboard working first, then spend an evening writing three or four alerts you will actually respond to. A dashboard tells you what happened when you go looking. An alert is what stops you needing to look.

Want this set up properly?

I build monitoring stacks for teams who would rather not spend a weekend on it — Prometheus and Grafana on bare servers or in Kubernetes, exporters for the things you actually run, alert rules tuned so they mean something, and dashboards that answer real questions rather than filling a screen.

If that sounds useful, you can hire me on Upwork.

The post Prometheus + Grafana From Scratch on One Server appeared first on John Nessime.