<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>VPS | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/vps/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/vps/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 31 Oct 2026 06:08:13 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>VPS | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/vps/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</title>
		<link>https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 31 Oct 2026 06:08:13 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Digital Forensics]]></category>
		<category><![CDATA[File Integrity Monitoring]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Persistence Mechanisms]]></category>
		<category><![CDATA[Rescue Mode]]></category>
		<category><![CDATA[Rootkits]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[systemd Timers]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Shell]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=496</guid>

					<description><![CDATA[<p>Killing the process removes the symptom. It does nothing about the SSH key on a service account or the timer pulling a fresh payload every twenty minutes. A working guide to compromised VPS recovery: how to judge the depth of an intrusion, keep the evidence, enumerate the access paths and persistence people miss, and decide honestly whether you can clean in place or need to rebuild.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email from the provider is short and unfriendly. Outbound scanning detected on port 22, network access to your instance has been rate limited, respond within 24 hours. You SSH in, find a process you don&#8217;t recognise chewing a core, kill it, delete the binary out of <code>/tmp</code>, and write back saying it&#8217;s handled.</p>



<p class="wp-block-paragraph">Three days later you get the same email.</p>



<p class="wp-block-paragraph">That gap is the whole problem with compromised VPS recovery. Killing the process removes the symptom. It does nothing about the SSH key that got added to a service account, the systemd timer that pulls a fresh payload every twenty minutes, or the reason they got in the first time. You cleaned up artifacts. You never closed the access.</p>



<p class="wp-block-paragraph">This post is about what you can honestly fix in place, what you can&#8217;t, and how to tell the difference before you sink a weekend into a box you&#8217;re going to rebuild anyway: evidence handling, the access paths and persistence people miss, why package verification proves less than it looks, and the one in-place technique that genuinely holds up.</p>



<h2 class="wp-block-heading">What compromised VPS recovery actually means</h2>



<p class="wp-block-paragraph">There are two goals hiding inside &#8220;recover the server&#8221; and they pull in different directions.</p>



<ul class="wp-block-list"><li><strong>Restore service.</strong> Get the site back, stop the abuse traffic, keep the client happy. Fast, and you can do it on the same instance.</li><li><strong>Restore trust.</strong> Be able to say, with something behind it, that nothing on this machine is executing code you didn&#8217;t put there. Slow, and on a machine where someone had root, often impossible.</li></ul>



<p class="wp-block-paragraph">A reinstall is not a technical superstition. It&#8217;s the cheapest way to buy the second goal. You throw away every byte you can&#8217;t account for and start from a base image plus config you control. Cleaning in place buys you the first goal quickly and the second one slowly, sometimes never.</p>



<p class="wp-block-paragraph">So the default answer is still: rebuild from a known-good image, restore validated data, re-point DNS. What follows is for when that isn&#8217;t available today, or when you have to investigate before you&#8217;re allowed to destroy anything.</p>



<h2 class="wp-block-heading">Decide this first: how deep did they get</h2>



<p class="wp-block-paragraph">Everything else follows from this. The depth of the compromise decides whether in-place recovery is defensible or wishful thinking.</p>



<h3 class="wp-block-heading">Unprivileged application code</h3>



<p class="wp-block-paragraph">A web shell dropped in a document root, running as the PHP-FPM pool user or <code>www-data</code>. It can read what that user reads, write where that user writes, and make outbound connections. It cannot touch <code>/usr/bin</code> or install a kernel module.</p>



<p class="wp-block-paragraph">In-place recovery is reasonable here, as long as you can prove the blast radius stayed inside that user. That proof is the hard part, not the cleanup.</p>



<h3 class="wp-block-heading">A service account with a path upward</h3>



<p class="wp-block-paragraph">Same as above, but the account has a sudoers entry, write access to a systemd unit directory, membership in the <code>docker</code> group, or a writable script that root runs on a schedule. Treat this as a root compromise until you&#8217;ve disproved it, because all four of those are one command away from root.</p>



<h3 class="wp-block-heading">Root</h3>



<p class="wp-block-paragraph">Now the machine can lie to you. Root can replace <code>ps</code>, edit the package database, truncate logs, and set the immutable attribute on files so your own cleanup silently fails. Anything the live system tells you is a claim, not a fact.</p>



<p class="wp-block-paragraph">You can still investigate. You cannot honestly certify the result. If someone needs a clean bill of health, rebuild.</p>



<h3 class="wp-block-heading">Kernel</h3>



<p class="wp-block-paragraph">A loadable kernel module or an eBPF program that hides processes, files and network connections from userspace. Your tools return whatever the rootkit decides they should return. There is no in-place recovery here, only offline analysis of the disk and a rebuild.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Before you touch anything, keep the evidence</h2>



<p class="wp-block-paragraph">The first fifteen minutes decide how much you&#8217;ll be able to learn later. Most of what tells you how bad it is only exists while the box is running, and most of it gets destroyed by the instinctive reaction.</p>



<ol class="wp-block-list"><li><strong>Isolate at the edge, not from inside.</strong> Use your provider&#8217;s firewall or security group to cut everything except your own source address. Running <code>ip link set eth0 down</code> from an SSH session locks you out and does not stop a local timer from running.</li><li><strong>Take a snapshot.</strong> Every serious VPS provider offers one, and it costs almost nothing to keep for a couple of weeks. This is your only chance at a copy of the disk in its compromised state. Providers like Hetzner, Contabo, OVHcloud and InterServer all expose snapshots or images from the control panel.</li><li><strong>Collect volatile state before you reboot.</strong> Sockets, processes, open files and the mapping from process to on-disk binary all disappear the moment the box restarts.</li><li><strong>Copy the collection off the box.</strong> To your workstation or a bucket, not to another directory on the same instance.</li></ol>



<p class="wp-block-paragraph">A collection pass that fits in a couple of minutes:</p>



<pre class="wp-block-code"><code># Sockets with owning process, numeric, TCP and UDP
ss -tunap &gt; /tmp/ir-sockets.txt

# Process tree, full command lines
ps auxf &gt; /tmp/ir-ps.txt

# Where each PID's executable actually lives on disk.
# A deleted binary still running shows up as "(deleted)".
for p in /proc/[0-9]*; do
  echo "$p -&gt; $(readlink "$p/exe" 2&gt;/dev/null)"
done &gt; /tmp/ir-exe.txt

# Logins, including failed ones
last -F -f /var/log/wtmp &gt; /tmp/ir-last.txt
lastb -F                 &gt; /tmp/ir-lastb.txt 2&gt;/dev/null

# Journal for the suspect window
journalctl --since "7 days ago" --no-pager &gt; /tmp/ir-journal.txt</code></pre>



<p class="wp-block-paragraph">The <code>readlink</code> loop is the one people skip and then wish they hadn&#8217;t. A process whose executable was unlinked after launch is a strong signal on its own, and it&#8217;s the only place the original path survives.</p>



<p class="wp-block-paragraph">If <code>auditd</code> was already running, pull its logs too. If it wasn&#8217;t, note that for the rebuild rather than installing it now on a machine you don&#8217;t trust.</p>



<h2 class="wp-block-heading">Failure family 1: you removed the payload and left the door</h2>



<p class="wp-block-paragraph">This is the one that produces the second abuse email. Access mechanisms are boring, they don&#8217;t show up in <code>top</code>, and they survive every cleanup that focuses on running processes.</p>



<pre class="wp-block-code"><code># Authorized keys for every account that has a home directory
for h in $(getent passwd | cut -d: -f6 | sort -u); do
  [ -f "$h/.ssh/authorized_keys" ] &amp;&amp; echo "== $h" &amp;&amp; cat "$h/.ssh/authorized_keys"
done

# sshd may have been told to read keys from somewhere else entirely
grep -Ei 'AuthorizedKeysFile|AuthorizedKeysCommand|PermitRootLogin|PasswordAuthentication|^Match' 
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2&gt;/dev/null

# Any account with UID 0 other than root
awk -F: '($3 == 0) {print}' /etc/passwd

# Accounts that can actually log in
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd

# Sudo grants, including drop-in files
cat /etc/sudoers; ls -la /etc/sudoers.d/</code></pre>



<p class="wp-block-paragraph">Two of these deserve more attention than they usually get. <code>AuthorizedKeysFile</code> pointing at a path outside a user&#8217;s home means your audit of <code>~/.ssh/authorized_keys</code> proves nothing. And a <code>Match</code> block near the bottom of <code>sshd_config</code> can quietly re-enable password authentication or root login for one user or one source address while the top of the file still looks locked down.</p>



<p class="wp-block-paragraph">Also check <code>/etc/pam.d/</code> for modules you don&#8217;t recognise. PAM is a legitimate extension point, which is exactly why it&#8217;s a good hiding place, and a single added line can accept any password for a given account.</p>



<h2 class="wp-block-heading">Failure family 2: persistence in the places you didn&#8217;t check</h2>



<p class="wp-block-paragraph">Cron gets checked. The rest usually doesn&#8217;t. MITRE ATT&amp;CK catalogues these as separate techniques for a reason: they&#8217;re independent, and finding one tells you nothing about the others.</p>



<pre class="wp-block-code"><code># Per-user crontabs and the system drop-in directories
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2&gt;/dev/null; done
ls -la /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /var/spool/cron/

# systemd: enabled units and every timer, including inactive ones
systemctl list-unit-files --type=service --state=enabled
systemctl list-timers --all
systemctl cat suspicious-unit.service   # shows the actual ExecStart

# Per-user systemd units survive logout when lingering is on
ls -la /root/.config/systemd/user/ /home/*/.config/systemd/user/ 2&gt;/dev/null
loginctl list-users

# One-shot at jobs, frequently overlooked
atq

# Dynamic linker hijacking
cat /etc/ld.so.preload 2&gt;/dev/null

# Shell and login hooks
ls -la /etc/profile.d/ /etc/update-motd.d/ 2&gt;/dev/null
grep -rn 'curl|wget|base64|/dev/tcp' /root/.bashrc /home/*/.bashrc 
  /etc/bash.bashrc 2&gt;/dev/null

# Unexpected setuid binaries and file capabilities
find / -xdev -perm -4000 -type f 2&gt;/dev/null
getcap -r / 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph"><code>/etc/ld.so.preload</code> is worth understanding rather than just grepping. Any library listed there is loaded into every dynamically linked process before anything else, which lets it intercept ordinary libc calls. That&#8217;s how a userspace rootkit makes <code>readdir</code> skip its own files and <code>ps</code> skip its own processes without touching the kernel. If that file exists on a machine where you didn&#8217;t put it, everything you&#8217;ve observed so far is suspect.</p>



<p class="wp-block-paragraph">The other one to internalise is timers. A cleanup that looks successful, followed by reinfection at a suspiciously round interval, is almost always a systemd timer or a cron entry you didn&#8217;t enumerate. <code>systemctl list-timers --all</code> matters because a timer that isn&#8217;t currently active is invisible without the flag.</p>



<h2 class="wp-block-heading">Failure family 3: verification that verifies less than you think</h2>



<p class="wp-block-paragraph">Package managers keep hashes of the files they installed, so you can ask the distro whether anything on disk drifted from what shipped.</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky, Fedora: verify every installed package
rpm -Va

# Debian and Ubuntu: dpkg's own verify, no extra package needed
dpkg -V

# Optional on Debian/Ubuntu, checks more files than dpkg -V
debsums -s</code></pre>



<p class="wp-block-paragraph">In <code>rpm -Va</code> output, the character positions are the checks that failed: <code>5</code> is a changed digest, <code>S</code> size, <code>T</code> timestamp, <code>M</code> mode. A <code>c</code> after the flags marks a config file, and config files change legitimately all the time. A modified digest on something in <code>/usr/bin</code> or <code>/usr/sbin</code> does not.</p>



<p class="wp-block-paragraph">Now the caveats, because this is where people talk themselves into a false clean bill of health:</p>



<ul class="wp-block-list"><li>It only covers files that came from a package. A web shell in your document root, a binary in <code>/tmp</code>, a rogue systemd unit and anything you compiled yourself are all invisible to it.</li><li>The database and the verifying binary both live on the machine being verified. Root can edit both.</li><li>Timestamps and sizes are trivially restorable. A digest change with an unchanged size and timestamp is a specific tell that someone tried to cover their tracks.</li></ul>



<p class="wp-block-paragraph">Treat findings as strong evidence and a clean result as weak evidence. That asymmetry is the useful part.</p>



<h3 class="wp-block-heading">Application-level checksums</h3>



<p class="wp-block-paragraph">For anything running WordPress, WP-CLI covers the gap the package manager leaves, because it verifies against checksums published by wordpress.org rather than anything stored locally:</p>



<pre class="wp-block-code"><code>wp core verify-checksums
wp plugin verify-checksums --all</code></pre>



<p class="wp-block-paragraph">Neither command covers themes or premium plugins that aren&#8217;t in the public repository, so pair them with a modification-time sweep of the document root.</p>



<h3 class="wp-block-heading">Rootkit scanners</h3>



<p class="wp-block-paragraph">The honest case for <code>rkhunter</code> and <code>chkrootkit</code>: they&#8217;re free, they run in a minute, and they catch known families and common mistakes that an experienced pair of eyes might skim past. Running one costs you nothing.</p>



<p class="wp-block-paragraph">The limit: they&#8217;re signature and heuristic based, they check for kits that were known when the signature set was written, and they ask the same possibly compromised kernel and libc that everything else asks. A clean scan means the obvious things aren&#8217;t there. It is not a negative result.</p>



<h2 class="wp-block-heading">Failure family 4: the way in is still open</h2>



<p class="wp-block-paragraph">If you finish the cleanup without an answer to &#8220;how did they get in&#8221;, you haven&#8217;t finished. You&#8217;ve just reset the clock.</p>



<p class="wp-block-paragraph">The practical technique is to find the earliest artifact you&#8217;re confident about, then read the access logs around that timestamp. Attackers rarely land quietly on the first attempt, and the noise before the successful request is usually more legible than the request itself.</p>



<pre class="wp-block-code"><code># Files in the document root modified in the last two weeks
find /var/www -xdev -type f -newermt '14 days ago' -printf '%T+ %pn' 2&gt;/dev/null | sort

# Same idea for the usual drop locations
find /tmp /var/tmp /dev/shm -xdev -type f -newermt '14 days ago' -ls 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph">The usual suspects, roughly in the order I check them: an outdated plugin or CMS component with a public exploit, an admin panel exposed to the whole internet, a weak or reused password on an account with a shell, a deploy key or CI token leaked into a repository, and a service bound to <code>0.0.0.0</code> that was only ever meant to be local. Redis and an unauthenticated Docker socket are the two that come up most in that last category.</p>



<p class="wp-block-paragraph">If your logs for the relevant window are gone, that absence is itself a finding. Shipping logs off-box to somewhere append-only, whether that&#8217;s a hosted platform or your own Loki instance, is the single change that would have made this investigation tractable. Worth noting for the rebuild.</p>



<h2 class="wp-block-heading">The in-place technique that actually holds up</h2>



<p class="wp-block-paragraph">Everything above shares a weakness: you&#8217;re asking a possibly compromised system to describe itself. There&#8217;s one way around that which doesn&#8217;t require a reinstall, and it&#8217;s the reason I don&#8217;t dismiss in-place work entirely.</p>



<p class="wp-block-paragraph">Boot the VPS into your provider&#8217;s rescue image, or attach a snapshot of the disk to a second clean instance. Then mount the target filesystem read-only and inspect it from a kernel the attacker never touched.</p>



<pre class="wp-block-code"><code># In the rescue environment: identify the target disk
lsblk -f

# Mount read-only. On ext4 add noload, on XFS add norecovery,
# so the kernel doesn't replay the journal and write to the disk.
mkdir -p /mnt/target
mount -o ro,noload /dev/vdb1 /mnt/target      # ext4
# mount -o ro,norecovery /dev/vdb1 /mnt/target  # xfs

# Inspect without chrooting into it
ls -la /mnt/target/etc/cron.d/
cat /mnt/target/etc/ld.so.preload 2&gt;/dev/null
find /mnt/target/etc/systemd/system -newermt '30 days ago' -ls</code></pre>



<p class="wp-block-paragraph">Why this is different in kind, not just degree: a userspace rootkit hooking libc, or a kernel module hiding directory entries, works by intercepting calls on the running system. Mounted from outside, those files are just bytes. Nothing gets to filter what you see.</p>



<p class="wp-block-paragraph">Two things to keep straight. Don&#8217;t <code>chroot</code> into the mounted filesystem and start running its binaries, because you&#8217;ve just handed control back to the attacker&#8217;s code. And read-only isn&#8217;t automatic: mounting a dirty ext4 or XFS filesystem will replay the journal and write to the device unless you pass the option that suppresses it, which is enough to invalidate the copy if anyone later cares about chain of custody.</p>



<h2 class="wp-block-heading">When your checks disagree</h2>



<ul class="wp-block-list"><li><strong>Package verification is clean but the behaviour continues.</strong> The payload isn&#8217;t in a packaged file. Look at the document root, <code>/opt</code>, <code>/usr/local</code>, and anything in a home directory. That&#8217;s most of the filesystem, and none of it is covered.</li><li><strong>A process has no matching binary on disk.</strong> The executable was unlinked after launch. Check <code>ls -l /proc/PID/exe</code> for the <code>(deleted)</code> suffix and the original path, then find what restarts it, because something must.</li><li><strong>A file refuses to be deleted as root.</strong> Check <code>lsattr</code> for the immutable attribute and clear it with <code>chattr -i</code>. Attackers use this specifically because it makes casual cleanup fail silently.</li><li><strong>Rebooting appears to fix it and it comes back on a schedule.</strong> Enumerate every timer and cron path again, including <code>systemctl list-timers --all</code> and per-user units under lingering sessions.</li><li><strong>Journal entries are missing for the interesting window.</strong> Run <code>journalctl --verify</code>. Truncation is a finding, not a dead end, and it argues for moving to the offline path.</li><li><strong>The outbound traffic continues after you&#8217;ve killed everything.</strong> You&#8217;re either looking at a hidden process or a second compromised container on the same host. Confirm from the provider&#8217;s traffic graphs rather than from tools on the box.</li></ul>



<h2 class="wp-block-heading">Arguments for staying in place that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;The rootkit scanner came back clean.&#8221;</strong> It checked for known kits using tools running on the suspect system. Absence of evidence, and weak absence at that.</li><li><strong>&#8220;I rotated all the passwords.&#8221;</strong> Passwords are one access path. SSH keys, API tokens, sudo rules, PAM modules and scheduled jobs are the others, and none of them care about your password reset.</li><li><strong>&#8220;It&#8217;s only a web shell, the OS is fine.&#8221;</strong> Sometimes true, and it&#8217;s the one case where in-place recovery is genuinely defensible. But it&#8217;s a conclusion you have to earn by ruling out privilege escalation, not an assumption you start from.</li><li><strong>&#8220;I restored from backup.&#8221;</strong> Restoring content doesn&#8217;t remove a key from <code>/root/.ssh</code>, and if the backup predates your detection by less than the dwell time, you&#8217;ve restored the compromise. Know when the earliest artifact appeared before you pick a restore point.</li><li><strong>&#8220;I don&#8217;t have time to rebuild.&#8221;</strong> The honest version of this is a real constraint, not a technical argument. But if your infrastructure is described in code and your data restore is tested, a rebuild is often faster than the second cleanup, never mind the third.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list"><li><strong>Snapshot first, always.</strong> Before any judgement call. It&#8217;s cheap and it&#8217;s the only irreversible decision in the whole process.</li><li><strong>Isolate at the provider edge</strong> and collect volatile state while the box is still up.</li><li><strong>Establish depth.</strong> Did anything run as root? If you can&#8217;t answer that with evidence, assume yes.</li><li><strong>If root is on the table, plan the rebuild now</strong> and treat the in-place work as investigation rather than remediation. The two goals stop competing once you&#8217;ve decided which one you&#8217;re pursuing.</li><li><strong>If it stayed inside one unprivileged account,</strong> clean in place: remove the payload, close the access paths, patch the entry point, rotate every credential that account could read.</li><li><strong>Do the offline read-only pass anyway.</strong> It&#8217;s twenty minutes in a rescue image and it&#8217;s the only evidence that doesn&#8217;t depend on the compromised machine&#8217;s cooperation.</li><li><strong>Rebuild the hardening on the new instance,</strong> not the old one. Key-only SSH, an allowlist on the admin port, automatic security updates, off-box logging, and file integrity monitoring with AIDE or a platform like Wazuh, so the next incident starts with a baseline instead of guesswork.</li></ol>



<p class="wp-block-paragraph">On that allowlist point: restricting SSH to a fixed source address is the single highest-value control here, and it&#8217;s the one people skip because their own IP changes. A VPN with a dedicated IP, which providers like NordVPN and Surfshark both offer, or a Cloudflare Tunnel in front of the admin path, gives you a stable origin to allow without exposing the port to the internet.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you ever recover a hacked Linux server without reinstalling?</h3>



<p class="wp-block-paragraph">Yes, when the compromise stayed inside an unprivileged application account and you can demonstrate that it did. Once anything ran as root, you can restore service in place but you cannot honestly certify the machine is clean, because every tool you&#8217;d use to check runs under the code you&#8217;re checking for.</p>



<h3 class="wp-block-heading">Should I reboot the VPS as soon as I notice something wrong?</h3>



<p class="wp-block-paragraph">Not first. A reboot destroys the process list, open sockets, and the mapping from running processes to deleted binaries, which is often the most informative evidence available. Snapshot, isolate at the edge, collect volatile state, then reboot if you need to.</p>



<h3 class="wp-block-heading">How long do attackers usually stay before I notice?</h3>



<p class="wp-block-paragraph">Long enough that you should be suspicious of any restore point close to your detection date. Rather than trusting a published average, find the earliest artifact you&#8217;re confident about on your own machine, then add margin. If your logs don&#8217;t reach back that far, that&#8217;s a finding in itself.</p>



<h3 class="wp-block-heading">Is restoring from a backup enough after a compromise?</h3>



<p class="wp-block-paragraph">Only if the backup predates the intrusion and covers everything that was modified. A content-only restore leaves SSH keys, cron entries, systemd units and package changes in place, so it usually removes the visible damage while preserving the access. Restore data onto a fresh instance instead of over a compromised one.</p>



<h3 class="wp-block-heading">What&#8217;s the fastest way to check for a compromised VPS?</h3>



<p class="wp-block-paragraph">Sockets with owning processes (<code>ss -tunap</code>), authorized keys across every home directory, every timer including inactive ones (<code>systemctl list-timers --all</code>), <code>/etc/ld.so.preload</code>, and package verification. That&#8217;s a few minutes and it catches the majority of commodity intrusions. It will not catch a competent one.</p>



<h3 class="wp-block-heading">Do I need to tell my hosting provider?</h3>



<p class="wp-block-paragraph">If they contacted you about abuse traffic, yes, and quickly, because unanswered abuse reports lead to suspension. It&#8217;s also worth asking whether they can supply traffic graphs for the affected period. That data comes from outside your instance, which makes it more trustworthy than anything on it.</p>



<h3 class="wp-block-heading">If there was personal data on the server, does that change anything?</h3>



<p class="wp-block-paragraph">It changes the order of operations. Preserve the snapshot before you clean anything, document what you did and when, and check your notification obligations before the technical work absorbs your attention. Cleanup that destroys evidence is a much bigger problem when the incident is reportable.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Compromised VPS recovery isn&#8217;t really a cleanup procedure. It&#8217;s a decision about what you&#8217;re able to claim afterwards.</p>



<p class="wp-block-paragraph">If the intrusion stayed inside an unprivileged account and you can show it, clean in place and move on. If root was ever involved, every check you run on that box is a statement from a machine with a motive. Take the snapshot, do the offline read-only pass because it&#8217;s the only view the attacker doesn&#8217;t control, and put the rebuilt hardening on a fresh instance rather than the one you&#8217;ve been arguing with.</p>



<p class="wp-block-paragraph">And find the way in. A cleanup that doesn&#8217;t end with a patched entry point isn&#8217;t a recovery, it&#8217;s an interval.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on a compromised server?</h2>



<p class="wp-block-paragraph">Incident work is unpleasant to do alone and easy to do in the wrong order. This is the kind of thing I help with:</p>



<ul class="wp-block-list"><li>Triage on a live VPS: establishing depth of compromise, collecting volatile evidence before it&#8217;s lost, and giving you a straight answer on clean-in-place versus rebuild</li><li>Offline analysis of a snapshot or rescue-mode disk, including timeline reconstruction from filesystem timestamps and whatever logs survived</li><li>Full enumeration of access paths and persistence: SSH configuration, PAM, sudoers, cron, systemd units and timers, preload hooks and setuid binaries</li><li>Finding the entry point rather than just the payload, correlating web and auth logs against the earliest artifacts on disk</li><li>Rebuilding the server properly afterwards, in code, with key-only SSH, an admin allowlist, off-box logging and file integrity monitoring in place from day one</li><li>WordPress and PHP application cleanup where the compromise came in through the stack rather than the OS</li></ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one right now, send me the output of <code>ss -tunap</code>, your <code>systemctl list-timers --all</code>, and the abuse report if you got one. That&#8217;s usually enough to tell you how deep this goes.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</title>
		<link>https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Decommissioning]]></category>
		<category><![CDATA[DirectAdmin]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[search-replace]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SPF]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Website Migration]]></category>
		<category><![CDATA[WordPress Hosting]]></category>
		<category><![CDATA[WordPress Migration]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=301</guid>

					<description><![CDATA[<p>Moving one WordPress site is easy. Moving twenty in a weekend is a pipeline, and the failures that cost you are the quiet ones: writes landing on the old server, cron firing on two boxes, certificates blocked by ACME rate limits. Here is the fleet migration process I use, from manifest to per-site rollback.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message arrived on Monday morning. Three quote requests submitted over the weekend, confirmed by the people who submitted them, and none of them in the CRM. The site was up. The contact form worked when I tested it. Nothing in the error log.</p>



<p class="wp-block-paragraph">Those submissions went to the old server. It was still answering for a slice of the internet twenty hours after the DNS change, and by then I had already stopped syncing it.</p>



<p class="wp-block-paragraph">That is the failure that separates a bulk WordPress migration from moving a single site. When you move one site you watch it. When you move twenty in a weekend, you are running a pipeline, and the things that go wrong are the ones nobody is watching: writes landing on a server you have mentally decommissioned, cron firing on two boxes at once, a certificate that never issued because a DNS record was still pointing at the old IP when the challenge ran.</p>



<p class="wp-block-paragraph">This is the process I use for fleet moves onto a single VPS: the manifest that makes the weekend repeatable, the two-pass sync, the database step people rush, the certificate limits that bite specifically at fleet scale, and a cutover order that keeps rollback cheap. It assumes SSH on both boxes and some comfort on the command line.</p>



<h2 class="wp-block-heading">The failure that actually costs you: writes landing on the old server</h2>



<p class="wp-block-paragraph">Every migration guide tells you to lower your TTL. Almost none of them tell you what happens in the gap afterwards, and the gap is where the money goes.</p>



<p class="wp-block-paragraph">Here is the mechanism. You change the A record. Most resolvers pick up the new IP inside the TTL window. Some do not: corporate resolvers that clamp minimum TTLs, ISP resolvers that ignore short values, browsers holding a connection open, a mail server that resolved the host an hour ago. For those clients, the old server is still the site. It still has a working database. It still accepts comments, form entries, orders, user registrations and password resets.</p>



<p class="wp-block-paragraph">Nothing errors. Nothing logs. The write succeeds on a database you are about to throw away.</p>



<p class="wp-block-paragraph">There are only two honest answers to this, and picking one per site is the first real decision of the weekend.</p>



<ul class="wp-block-list">
<li><strong>Accept the loss.</strong> Fine for brochure sites, documentation, most blogs. A missed comment is not a business problem.</li>
<li><strong>Make the old server read-only the moment DNS changes.</strong> Mandatory for anything with a checkout, a booking flow, a members area or a lead form that feeds a sales process.</li>
</ul>



<p class="wp-block-paragraph">Read-only in practice means maintenance mode on the old copy, a static holding page on the old vhost, or pointing the old install&#8217;s <code>wp-config.php</code> at a database user with SELECT only. Any of those turns silent data loss into a visible error the visitor can react to. Someone who sees &#8220;temporarily unavailable&#8221; comes back. Someone whose order vanished does not.</p>



<p class="wp-block-paragraph">Do not skip this because the old server &#8220;will be off soon&#8221;. The window between DNS change and old server shutdown is the longest, least supervised part of the whole weekend.</p>



<h2 class="wp-block-heading">Build the fleet manifest before you touch a server</h2>



<p class="wp-block-paragraph">Twenty sites is the point where memory stops working. You will not remember which one runs an old PHP branch, which one has a hardcoded absolute path in a theme, which one has MX records pointing somewhere else entirely.</p>



<p class="wp-block-paragraph">So the first artifact is a table. One row per site, filled in before anything moves.</p>



<ul class="wp-block-list">
<li>Document root and database name</li>
<li>Current <code>siteurl</code> and <code>home</code> values, exactly as stored</li>
<li>PHP version the site currently runs under</li>
<li>Disk size of the document root and dump size of the database</li>
<li>Where DNS is authoritative, and who holds the login</li>
<li>Whether mail for that domain is on this server or elsewhere</li>
<li>Does it take writes from the public: comments, forms, checkout, registration</li>
<li>Cache and object cache in use</li>
<li>Cutover batch number and rollback owner</li>
</ul>



<p class="wp-block-paragraph">Most of that you can pull off the old server rather than typing it. This walks a DirectAdmin-style layout and prints a tab-separated line per WordPress install:</p>



<pre class="wp-block-code"><code>for d in /home/*/domains/*/public_html; do
  [ -f "$d/wp-config.php" ] || continue
  printf '%st%st%sn' 
    "$d" 
    "$(wp --path="$d" option get siteurl --skip-plugins --skip-themes 2&gt;/dev/null)" 
    "$(du -sh "$d" | cut -f1)"
done</code></pre>



<p class="wp-block-paragraph">The <code>--skip-plugins --skip-themes</code> flags matter here. Without them WP-CLI boots the full site, and one broken plugin on one install will stop your inventory halfway through with a fatal error.</p>



<p class="wp-block-paragraph">The DNS column is the one people fill in last and regret. Check where each zone actually lives before the weekend, not during it. A domain whose nameservers point at Cloudflare while you are editing records in the old host&#8217;s DNS panel will not move no matter how many times you refresh.</p>



<h2 class="wp-block-heading">Rebuild the stack, don&#8217;t clone the disk</h2>



<p class="wp-block-paragraph">There is a tempting shortcut where you image the old server and restore it onto the new one. It usually works and it is usually the wrong call, because you have just carried across every accumulated fix, every orphaned config, and every reason the old box got slow enough to need replacing.</p>



<p class="wp-block-paragraph">Build the destination clean. Whatever the provider is, whether that is Contabo, InterServer, Hetzner or a machine you already have, the sequence is the same: base OS, web server, PHP with the version and extensions the manifest says you need, database server, control panel if you use one, firewall, then a hardening pass.</p>



<p class="wp-block-paragraph">Two things to get right before any files land.</p>



<p class="wp-block-paragraph"><strong>PHP versions.</strong> If three sites need an older branch and seventeen are happy on current, install both and make the per-site assignment explicit in the manifest. Discovering the mismatch after cutover means a white screen on a live domain instead of a note in a spreadsheet. Check the extension list too, not just the version number, because <code>imagick</code>, <code>soap</code> and <code>intl</code> are the usual absentees.</p>



<p class="wp-block-paragraph"><strong>Access during the weekend.</strong> If you lock SSH and the control panel behind an IP allowlist, decide beforehand what your source address will be. A rotating home IP is how people lock themselves out of their own migration at two in the morning. A dedicated-IP option from a VPN provider such as NordVPN or Surfshark gives you one stable address to allowlist; a second admin path on a non-standard port works too. Deciding at the moment you are locked out does not.</p>



<h2 class="wp-block-heading">Two-pass rsync beats one big copy</h2>



<p class="wp-block-paragraph">The mistake at fleet scale is treating the file copy as a cutover-day task. Copy everything on cutover day and your maintenance window is however long twenty document roots take, which for a media-heavy fleet is hours you do not have.</p>



<p class="wp-block-paragraph">Split it. Do a warm sync days in advance while both sites are live, then a delta sync at cutover that moves only what changed.</p>



<p class="wp-block-paragraph">Warm sync, run from the old server, no deletes:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph">What those flags buy you: <code>-a</code> preserves permissions, times, symlinks and ownership; <code>-H</code> keeps hard links, which some backup plugins and dedup schemes rely on; <code>-A</code> and <code>-X</code> carry ACLs and extended attributes; <code>--numeric-ids</code> stops rsync remapping ownership through name lookups when the UIDs differ between boxes, which is exactly what happens when the new server creates users in a different order.</p>



<p class="wp-block-paragraph">Delta sync at cutover, same command with one addition:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --delete --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph"><code>--delete</code> removes files on the destination that no longer exist on the source. It is what makes the second pass an accurate mirror rather than a merge of two states. It is also the flag that will destroy your afternoon if you get the paths wrong, so check the trailing slashes: a source path ending in a slash copies the contents of the directory, one without it copies the directory itself, one level deeper than you meant.</p>



<p class="wp-block-paragraph">Run the delta pass once with <code>--dry-run</code> before running it for real. And on SELinux systems, relabel the tree afterwards: files arriving over rsync carry contexts the web server may not accept, and that produces a 403 which has nothing to do with permission bits.</p>



<h2 class="wp-block-heading">Databases last, and never with sed</h2>



<p class="wp-block-paragraph">Files can be stale for a few days. Databases cannot. Dump each database as late as you reasonably can, immediately before that site&#8217;s cutover, not on Friday for a Sunday move.</p>



<pre class="wp-block-code"><code># on the old server, per site
wp --path=/home/user/domains/example.com/public_html db export 
  /root/migration/example.com.sql --skip-plugins --skip-themes</code></pre>



<p class="wp-block-paragraph">Import on the new box, then handle URLs. If the domain is not changing, you may have nothing to replace. If the site is moving from HTTP to HTTPS, or from a staging hostname, or from a temporary preview domain you used for testing, you do.</p>



<p class="wp-block-paragraph">The rule that matters: do not run a find-and-replace over the SQL file with <code>sed</code>. WordPress stores widget settings, theme options, page builder layouts and plugin config as PHP serialized arrays, and every serialized string carries a byte-length prefix. Change the string without recalculating the prefix and the array fails to unserialize. The site does not error loudly. It loses its widgets, or its theme options revert to defaults, or the page builder renders an empty page.</p>



<p class="wp-block-paragraph">WP-CLI unserializes, replaces and reserializes, which is why it is the only tool I use for this.</p>



<pre class="wp-block-code"><code>wp search-replace 'http://example.com' 'https://example.com' 
  --all-tables-with-prefix 
  --precise 
  --skip-columns=guid 
  --report-changed-only 
  --dry-run</code></pre>



<p class="wp-block-paragraph">Read the dry-run output before you commit. Each flag earns its place: <code>--all-tables-with-prefix</code> catches plugin tables that share your prefix and would otherwise be skipped, <code>--precise</code> forces a full unserialize pass on every column instead of the faster string-level path, and <code>--skip-columns=guid</code> leaves the GUID column alone because feed readers use it as an identifier, not as a URL to fetch. Drop <code>--dry-run</code> to apply.</p>



<p class="wp-block-paragraph">Flush the object cache afterwards if the site uses one. A stale Redis or Memcached entry will happily serve you the old values and convince you the replacement failed.</p>



<h2 class="wp-block-heading">TLS at fleet scale: where Let&#8217;s Encrypt rate limits actually bite</h2>



<p class="wp-block-paragraph">Twenty certificates in one weekend is where a limit you have never noticed becomes the thing that blocks you.</p>



<p class="wp-block-paragraph">Let&#8217;s Encrypt publishes its limits and they are worth reading properly before a fleet move. Two of them are the ones that catch migrations.</p>



<ul class="wp-block-list">
<li><strong>New certificates per registered domain.</strong> Up to 50 certificates per registered domain every 7 days, counted globally across all accounts. If your twenty sites are twenty separate registered domains, you are nowhere near it. If they are subdomains of one or two parent domains, which is common for an agency fleet, you can burn through it.</li>
<li><strong>Authorization failures per identifier per account.</strong> Up to 5 failures per hostname per account per hour, refilling at one every 12 minutes. This is the one that actually bites, because it is triggered by the exact thing you are doing: requesting a certificate for a hostname whose DNS still points at the old server, so the HTTP-01 challenge is served by the wrong machine and fails.</li>
</ul>



<p class="wp-block-paragraph">Order matters as a result. If you issue before DNS moves, HTTP-01 fails. If you issue after DNS moves, visitors hit a certificate error in the gap. Three ways out, in the order I reach for them:</p>



<ol class="wp-block-list">
<li><strong>DNS-01 validation.</strong> Proves control through a TXT record instead of an HTTP request, so it does not care where the A record points. You can issue every certificate before touching DNS. This is the clean answer if your DNS provider has an API your ACME client supports.</li>
<li><strong>Cut DNS first, issue immediately after.</strong> Simple, and the exposure is short if you are watching. Batch it so you are never waiting on twenty challenges at once.</li>
<li><strong>Copy the existing certificates and keys across.</strong> Buys you a bridge until the new server can renew on its own. Only viable if you control the old server&#8217;s filesystem and the certificates have real time left.</li>
</ol>



<p class="wp-block-paragraph">Whichever you choose, test the plumbing against the staging environment first. Most ACME clients expose this as a dry-run mode, and it exists precisely so you can fail as many times as you like without spending production budget.</p>



<p class="wp-block-paragraph">One more thing that catches people: if any site has HSTS enabled with a long max-age, browsers that have seen it will refuse to fall back to HTTP. There is no grace period and no way to talk them out of it. Those sites need a valid certificate on the new server from the first request, which pushes them firmly toward DNS-01 or a copied certificate.</p>



<h2 class="wp-block-heading">Planning the cutover order for a bulk WordPress migration</h2>



<p class="wp-block-paragraph">Do not move twenty sites at once. Batch them, and batch by risk rather than alphabetically or by size.</p>



<ol class="wp-block-list">
<li><strong>Batch one: the throwaways.</strong> Two or three low-traffic sites you own or that nobody would notice. This batch exists to prove the stack, not to make progress.</li>
<li><strong>Batch two: the ordinary majority.</strong> Standard brochure and content sites, no checkout, no logins that matter.</li>
<li><strong>Batch three: the ones with writes.</strong> Stores, membership sites, anything with a form feeding a sales process.</li>
<li><strong>Batch four: the awkward ones.</strong> Custom stacks, odd PHP requirements, third-party integrations that whitelist your server IP.</li>
</ol>



<p class="wp-block-paragraph">Stop between batches. Look at the error logs on the new server, not just the homepages. Twenty sites failing the same way is one problem; catching it after three is a fifteen-minute fix, catching it after twenty is a rollback.</p>



<p class="wp-block-paragraph">The per-site sequence inside a batch:</p>



<ol class="wp-block-list">
<li>Drop the TTL on that domain&#8217;s records to a low value, at least a day ahead.</li>
<li>Run the delta rsync.</li>
<li>Dump, transfer and import the database.</li>
<li>Run search-replace if the URL is changing at all.</li>
<li>Test through a hosts-file entry or a resolved request, before DNS moves.</li>
<li>Change the A and AAAA records.</li>
<li>Put the old copy read-only.</li>
<li>Verify from outside, then move to the next site.</li>
</ol>



<p class="wp-block-paragraph">Step five is the one people skip and should not. You can send a request to the new server as if DNS had already changed, without changing anything:</p>



<pre class="wp-block-code"><code>curl -sSI --resolve example.com:443:NEW_SERVER_IP https://example.com/
curl -sS --resolve example.com:443:NEW_SERVER_IP https://example.com/wp-login.php | head -n 20</code></pre>



<p class="wp-block-paragraph">That gives you real status codes, real redirects and a real TLS handshake against the new box while the live site is untouched. If that returns a 301 loop or a 500, you have found it before any visitor did.</p>



<h2 class="wp-block-heading">Cron, email, and the things that run twice</h2>



<p class="wp-block-paragraph">During the overlap window you have two live copies of every site. Both have a database. Both have a scheduler.</p>



<p class="wp-block-paragraph">WordPress schedules through <code>wp-cron.php</code>, fired by visitor traffic. The old server still gets traffic, so it still fires cron, so it still sends the newsletter, still processes the subscription renewal, still runs the backup plugin. Subscribers get things twice. Worse, the site&#8217;s system crontab came across in your file sync and is now also running on the new box.</p>



<p class="wp-block-paragraph">Handle it explicitly. On the old copy, disable the loopback scheduler when you make it read-only:</p>



<pre class="wp-block-code"><code>define( 'DISABLE_WP_CRON', true );</code></pre>



<p class="wp-block-paragraph">And on the new server, set up a real system cron per site rather than leaving it to traffic, because a site with no traffic yet has no scheduler at all:</p>



<pre class="wp-block-code"><code>wp --path=/home/user/domains/example.com/public_html cron event run --due-now</code></pre>



<p class="wp-block-paragraph">Wrap that in a cron entry or a systemd timer at whatever interval the site needs, and check <code>wp cron event list</code> before assuming nothing important is queued.</p>



<p class="wp-block-paragraph">Email is the parallel problem and it fails quietly in the other direction. Your sites now send from a new IP. If SPF still authorizes only the old server&#8217;s address, and DKIM keys did not come across, and DMARC is set to anything stricter than none, your transactional mail starts landing in spam. Nothing bounces visibly. Password resets just stop arriving and you hear about it in support tickets a week later.</p>



<p class="wp-block-paragraph">Three records to check per domain. MX is the one not to touch casually: if mail lives elsewhere and you replace a zone wholesale, the mailboxes go with it.</p>



<ul class="wp-block-list">
<li><strong>MX:</strong> unchanged unless mail is genuinely moving with you.</li>
<li><strong>SPF:</strong> add the new server&#8217;s IP before cutover, remove the old one after the overlap window closes, not during.</li>
<li><strong>DKIM:</strong> keys are per-server. Either copy them or regenerate and publish the new public key.</li>
</ul>



<p class="wp-block-paragraph">The more resilient answer is to stop sending through the server at all. Routing WordPress mail through an authenticated SMTP relay means outbound deliverability stops depending on which machine the site happens to live on, which makes the next migration a non-event.</p>



<h2 class="wp-block-heading">Troubleshooting the weekend</h2>



<p class="wp-block-paragraph"><strong>Redirect loop on the new server.</strong> Usually <code>siteurl</code> and <code>home</code> disagreeing with what the vhost or reverse proxy is doing about HTTPS. Check both values with <code>wp option get siteurl</code> and <code>wp option get home</code>. If the site sits behind a proxy or CDN that terminates TLS, WordPress may see HTTP on the back end and redirect forever.</p>



<p class="wp-block-paragraph"><strong>Site loads, media does not.</strong> Either uploads did not finish syncing, ownership is wrong, or database paths still reference the old host. Compare a file listing on both sides before assuming it is a database problem.</p>



<p class="wp-block-paragraph"><strong>Widgets or theme options empty after import.</strong> This is the serialization damage described earlier. It is not recoverable by re-running the replacement. Restore the database from the dump and redo the URL change with WP-CLI.</p>



<p class="wp-block-paragraph"><strong>Certificate issuance failing repeatedly.</strong> Confirm what the challenge is actually hitting before retrying, because each failure spends part of your hourly authorization budget.</p>



<pre class="wp-block-code"><code>dig +noall +answer example.com A
dig +noall +answer @1.1.1.1 example.com A
openssl s_client -connect NEW_SERVER_IP:443 -servername example.com &lt; /dev/null 2&gt;/dev/null | openssl x509 -noout -subject -dates</code></pre>



<p class="wp-block-paragraph">The first query uses your local resolver, the second asks a public one directly, and the difference between them tells you whether you are looking at a real propagation state or your own DNS cache.</p>



<p class="wp-block-paragraph"><strong>Site fine, admin slow.</strong> Often a plugin trying to reach an external service that whitelisted the old IP. Look for API integrations, license checks and payment gateways in the manifest&#8217;s awkward-ones column.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Lowering TTL on cutover day. The old value is still cached; the change has no effect for as long as the old TTL says.</li>
<li>Cancelling the old server as soon as the sites load. Keep it running, read-only, for the whole overlap plus a margin. It is your rollback.</li>
<li>Replacing an entire DNS zone instead of editing the A record, and taking MX, SPF and verification TXT records with it.</li>
<li>Running search-replace before importing, or on the SQL file instead of the live database.</li>
<li>Testing only the homepage. Homepages are static and cached. Test a login, a form submission, an admin page and an uploaded image.</li>
<li>Leaving both schedulers live and finding out through duplicate customer emails.</li>
<li>Assuming the new server&#8217;s PHP extension set matches the old one because the version number does.</li>
<li>Treating rollback as a fleet-level decision. Per-site rollback is what keeps one bad site from stalling nineteen good ones.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a fleet move</h2>



<ul class="wp-block-list">
<li>Write the manifest first and update it as you go. It is the migration record and the handover document.</li>
<li>Take a restorable backup of each site to somewhere off both servers before you start, and test restoring one. A backup you have not restored is a hypothesis.</li>
<li>Warm sync early, delta sync at cutover, dump databases last. Batch by risk, and prove the stack on sites nobody will miss.</li>
<li>Verify every site from outside with a resolved request before its DNS moves.</li>
<li>Point uptime monitoring at the new IP before cutover, not after. UptimeRobot or Better Stack will tell you about a broken site faster than the client will.</li>
<li>Make the old copy read-only at the moment of DNS change, not at the end of the weekend.</li>
<li>Keep a rollback note per site: which records changed, what they were before, who can revert them.</li>
<li>Re-check cache configuration after the move rather than assuming it carried over, whether that is WP Rocket, WP Fastest Cache or a server-level layer.</li>
<li>Retire the old server on a scheduled date with a final archived backup, not by forgetting to renew it.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a bulk WordPress migration of 20 sites realistic in one weekend?</h3>



<p class="wp-block-paragraph">Yes, if the preparation happens in the week before. The weekend itself should be delta syncs, database moves and DNS changes. If you are still installing PHP extensions on Saturday morning, the weekend is not long enough.</p>



<h3 class="wp-block-heading">How low should I set the TTL, and how far ahead?</h3>



<p class="wp-block-paragraph">A low value like 300 seconds, set at least 24 to 48 hours before cutover so the previous TTL has fully expired everywhere. Going much lower than 300 buys little and increases query load. Raise it back after the migration settles.</p>



<h3 class="wp-block-heading">Will moving hosts hurt my search rankings?</h3>



<p class="wp-block-paragraph">Changing the server behind an unchanged domain is not itself a ranking event. What hurts is what goes wrong during it: 5xx responses while crawlers visit, failed certificates, redirect loops, a staging copy left indexable. Keep URLs identical, keep the site responding, and check Search Console afterwards.</p>



<h3 class="wp-block-heading">Can I use a migration plugin instead of the command line for this?</h3>



<p class="wp-block-paragraph">For one site, plugins are a reasonable choice and handle serialized data correctly. At twenty the arithmetic changes: twenty manual export and import cycles through a browser, with no dry run and no delta sync. The command line is repeatable and scriptable, which is the point at fleet scale.</p>



<h3 class="wp-block-heading">How long should I keep the old VPS running?</h3>



<p class="wp-block-paragraph">At least a week after the last site cuts over, in read-only mode. Some problems only appear over a full traffic cycle: weekly scheduled jobs, integrations that call in periodically, mail arriving at the old MX. Keep an archived backup after you destroy it.</p>



<h3 class="wp-block-heading">What if one site breaks after cutover?</h3>



<p class="wp-block-paragraph">Revert that site&#8217;s A record and lift the read-only restriction on the old copy. That is why the old server stays up and why rollback is per site. If you sync the database back afterwards, be careful not to overwrite writes that landed on the new server in the meantime.</p>



<h3 class="wp-block-heading">Do I need to move email at the same time?</h3>



<p class="wp-block-paragraph">No, and separating them is usually wiser. Move web first, confirm it is stable, then plan mailbox migration as its own change with its own window. Combining them means a single mistake in one DNS zone takes down both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A bulk WordPress migration is not twenty small migrations. It is one process run twenty times, and the difference is that at scale the visible failures get caught while the invisible ones do not.</p>



<p class="wp-block-paragraph">The copy is the easy part. rsync and WP-CLI are reliable, well documented and boring. What costs you is the overlap window: two servers both answering, both writing, both scheduling, for hours after you have mentally finished. Decide before the weekend what happens to writes on the old server, make that decision per site, and act on it at the moment DNS changes rather than at the end.</p>



<p class="wp-block-paragraph">Do that, batch by risk, and keep a rollback that works at the level of a single site, and twenty sites in a weekend is an ordinary piece of work rather than a gamble.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help moving a fleet of sites?</h2>



<p class="wp-block-paragraph">Migrations are a good thing to have a second pair of hands on, because the expensive mistakes are the quiet ones. This is work I do regularly:</p>



<ul class="wp-block-list">
<li>Auditing an existing fleet and producing the migration manifest, batch order and rollback plan before anything moves</li>
<li>Provisioning and hardening the destination VPS, including multiple PHP versions, control panel setup and firewall rules</li>
<li>Running the sync and database cutover with WP-CLI, including URL replacement on serialized data</li>
<li>Planning TLS issuance around ACME rate limits, including DNS-01 validation and automated renewal</li>
<li>Sorting out post-migration email: SPF, DKIM, DMARC and moving transactional mail onto an authenticated relay</li>
<li>Sitting in as the on-call engineer for the cutover window itself, or reviewing a plan you have already written</li>
</ul>



<p class="wp-block-paragraph">If you are planning a move and something in it worries you, send me the concrete thing: the site list, the current vhost config, the rsync output, the certbot error. I would rather look at the actual output than talk in general terms.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users</title>
		<link>https://john-nessime.com/blog/technical-guides/nginx-rate-limiting-bot-mitigation/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 27 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Networking]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Bot Mitigation]]></category>
		<category><![CDATA[CDN]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[DDoS Protection]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[limit_req]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Reverse Proxy]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=256</guid>

					<description><![CDATA[<p>A limit_req block that parses cleanly can still be doing absolutely nothing, or throttling every visitor behind one CDN edge. This guide covers Nginx rate limiting and bot mitigation as they actually behave in production: getting the key right behind a proxy, what burst and nodelay do to your traffic shape, sizing zones correctly, rolling out in dry run mode, and layering bot handling from robots.txt through user agent maps to firewall escalation.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/nginx-rate-limiting-bot-mitigation/">Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;site is slow, nothing changed.&#8221; Load average was climbing, PHP-FPM workers were all busy, and the access log showed the same handful of URLs being hit over and over: search queries with random terms, paginated archives forty pages deep, and a slow crawl through every tag page on the site. Nothing was down. Everything was just late.</p>



<p class="wp-block-paragraph">The fix looked obvious. Drop in a <code>limit_req_zone</code>, apply it, reload, done. Except the reload changed nothing at all, and the reason took an embarrassingly long time to find.</p>



<p class="wp-block-paragraph">This post covers Nginx rate limiting and bot mitigation the way it actually behaves in front of a real site: how the limiting modules key requests, what <code>burst</code> and <code>nodelay</code> really do to your traffic shape, why a limit that looks correct in the config can be doing nothing (or throttling everyone at once), and how to layer bot handling so you are not playing whack-a-mole with user agent strings forever.</p>



<h2 class="wp-block-heading">The failure that hides: your limit is keyed on the wrong address</h2>



<p class="wp-block-paragraph">This is the one that bites hardest, because the config parses, <code>nginx -t</code> passes, and the error log stays quiet. Nothing tells you it is wrong.</p>



<p class="wp-block-paragraph">If anything sits in front of Nginx (Cloudflare, another CDN, a load balancer, or a local Varnish tier), then <code>$remote_addr</code> is the address of that proxy, not the visitor. Key a rate limit on it and you get one of two outcomes, both bad:</p>



<ul class="wp-block-list">
<li>Every visitor arriving through a given edge node shares a single bucket. One aggressive client burns the budget and legitimate users start collecting 503s.</li>

<li>Or your rate is generous enough that the shared bucket never fills, in which case the limit is decorative and the abusive client sails straight through.</li>
</ul>



<p class="wp-block-paragraph">The real IP module fixes this, but only if you trust the right sources. It rewrites <code>$remote_addr</code> and <code>$binary_remote_addr</code> from a header, and only for connections coming from addresses you have explicitly declared trustworthy.</p>



<pre class="wp-block-code"><code># http context. One entry per CDN range you actually receive traffic from.
set_real_ip_from 203.0.113.0/24;
set_real_ip_from 2001:db8::/32;

real_ip_header CF-Connecting-IP;   # or X-Forwarded-For for a generic proxy
real_ip_recursive on;</code></pre>



<p class="wp-block-paragraph">Two things matter here more than the syntax.</p>



<p class="wp-block-paragraph">First, never widen <code>set_real_ip_from</code> to <code>0.0.0.0/0</code>. That tells Nginx to believe a client-supplied header from anyone, which means an attacker can pick a new &#8220;client IP&#8221; on every request and your per-IP limit becomes unlimited. It also poisons your logs, your <code>deny</code> rules and anything downstream that trusts the forwarded address.</p>



<p class="wp-block-paragraph">Second, after the real IP module has run, <code>$binary_remote_addr</code> holds the visitor address and <code>$realip_remote_addr</code> holds the original connecting address (the proxy). Key your limits on <code>$binary_remote_addr</code>. Keying on <code>$realip_remote_addr</code> puts you right back in the shared-bucket failure, silently.</p>



<p class="wp-block-paragraph">The proxy ranges change over time, so treat the trust list as a file that gets regenerated on a schedule and reloaded, not something you paste in once. On a self-managed VPS from a provider like Contabo or InterServer, a small cron job that fetches the current ranges, rewrites an include file, runs <code>nginx -t</code>, and only reloads on success is about twenty lines of shell and saves you from a very confusing afternoon later.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What limit_req actually does, and what burst really means</h2>



<p class="wp-block-paragraph">Nginx rate limiting for requests uses a leaky bucket. The zone stores, per key, the current number of excessive requests. The rate defines how fast that excess drains. Two directives do the work: <code>limit_req_zone</code> declares the bucket in the <code>http</code> context, <code>limit_req</code> applies it in <code>http</code>, <code>server</code> or <code>location</code>.</p>



<pre class="wp-block-code"><code>http {
    limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m   rate=20r/m;

    limit_req_status 429;
    limit_req_log_level warn;
}</code></pre>



<p class="wp-block-paragraph">The rate is expressed in requests per second (<code>r/s</code>) or requests per minute (<code>r/m</code>). If you want less than one request per second, use <code>r/m</code>. There is no native per-hour or per-day unit in the stock module, which surprises people writing password-reset limits.</p>



<h3 class="wp-block-heading">burst without nodelay is a queue, not a rejection</h3>



<p class="wp-block-paragraph">This is the second-most common misunderstanding after the keying problem. By default, excessive requests are <em>delayed</em> so that they come out at the configured rate. They are only rejected once the number waiting exceeds <code>burst</code>.</p>



<pre class="wp-block-code"><code>location /search/ {
    limit_req zone=general burst=20;
}</code></pre>



<p class="wp-block-paragraph">At 10r/s, a client that fires 21 requests at once gets one served immediately, nineteen held and released over roughly two seconds, and the last one rejected. Those nineteen held requests are still open connections on your server. You have not reduced concurrency, you have converted a spike into a slow trickle and paid for it in worker slots and file descriptors.</p>



<p class="wp-block-paragraph">That is genuinely useful for smoothing traffic to a fragile upstream. It is a bad idea for abuse control, where holding the connection is exactly what the attacker wants.</p>



<h3 class="wp-block-heading">nodelay, delay, and picking between them</h3>



<p class="wp-block-paragraph"><code>nodelay</code> changes the behaviour to: serve everything in the burst window immediately, reject everything past it. The bucket still drains at the configured rate, so a client that sustains the burst rate will hit the wall and stay there.</p>



<pre class="wp-block-code"><code>location /wp-login.php {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    # ...fastcgi_pass or proxy_pass here
}</code></pre>



<p class="wp-block-paragraph">The <code>delay</code> parameter is the middle ground: it sets the point within the burst at which requests start being delayed rather than served immediately. <code>burst=20 delay=10</code> means the first ten excess requests go straight through, the next ten get queued, and anything beyond is rejected.</p>



<p class="wp-block-paragraph">How I decide: if a human clicking around your site could plausibly generate the spike, use <code>nodelay</code> or a <code>delay</code> that covers it. A single page load pulling twelve uncached assets through Nginx is one &#8220;request&#8221; to the user and twelve to the limiter. If the endpoint is machine-facing and expensive (search, export, login, an API write path), tighten the rate, keep the burst small, and use <code>nodelay</code> so rejection is fast and cheap.</p>



<p class="wp-block-paragraph">Also set <code>limit_req_status 429</code>. The default is 503, which tells clients and monitoring that <em>your server</em> is broken rather than that <em>they</em> are being throttled. 429 is the correct semantic, and well-behaved clients back off on it.</p>



<h3 class="wp-block-heading">Sizing the zone, and the number most guides get wrong</h3>



<p class="wp-block-paragraph">You will see &#8220;10m holds about 160,000 IP addresses&#8221; repeated everywhere. That is the 32-bit figure. Nginx documents the stored state for <code>limit_req</code> as 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms, so one megabyte holds roughly 16 thousand states on 32-bit and roughly 8 thousand on 64-bit. On the 64-bit server you are almost certainly running, a 10m zone is closer to 80,000 states.</p>



<p class="wp-block-paragraph">This matters because of what happens when the zone fills. Nginx evicts the least recently used state. If it still cannot allocate, the request is terminated with your configured error status. A zone that is constantly thrashing under a distributed crawl will start returning errors for reasons that have nothing to do with the rate you configured. If you are limiting a high-cardinality key, give the zone room.</p>



<h2 class="wp-block-heading">limit_conn solves a different problem, and you probably need both</h2>



<p class="wp-block-paragraph"><code>limit_req</code> caps how fast requests arrive. It does nothing about a client that opens sixty connections and reads each response one byte at a time. That is a concurrency problem, and it belongs to <code>limit_conn</code>.</p>



<pre class="wp-block-code"><code>http {
    limit_conn_zone $binary_remote_addr zone=perip:10m;
    limit_conn_status 429;
}

server {
    limit_conn perip 20;

    location /downloads/ {
        limit_conn perip 4;
        limit_rate_after 2m;
        limit_rate 512k;
    }
}</code></pre>



<p class="wp-block-paragraph">Three behaviours worth knowing before you pick a number.</p>



<ul class="wp-block-list">
<li>Not every connection counts. Nginx only counts a connection once the full request header has been read and the request is being processed. A client stalling mid-header is not counted here, which is why <code>client_header_timeout</code> and <code>client_body_timeout</code> still matter.</li>

<li>Under HTTP/2 and HTTP/3, each concurrent request counts as a separate connection. A limit of 4 that felt generous on HTTP/1.1 can break a modern browser loading a page with many assets. This is a very common self-inflicted outage.</li>

<li>The <code>limit_conn</code> state is smaller than the <code>limit_req</code> state, so the same zone size holds roughly twice as many entries.</li>
</ul>



<p class="wp-block-paragraph">The pairing is the point: <code>limit_req</code> protects against volume, <code>limit_conn</code> protects against occupancy. Rate limiting alone leaves you open to slow-read exhaustion, and connection limiting alone does nothing about a client making one fast request after another on a keepalive connection.</p>



<h2 class="wp-block-heading">Choosing keys: IP is a starting point, not the answer</h2>



<p class="wp-block-paragraph">The key can be any variable or combination of variables. That flexibility is where most of the value is, because per-IP limits have a real accuracy problem in both directions.</p>



<p class="wp-block-paragraph">On one side, carrier-grade NAT, corporate egress, university networks and shared VPN exit nodes (NordVPN, Surfshark and every other provider works this way) put many genuine users behind one address. A tight per-IP limit will hit them. On the other side, a distributed scraper renting a few hundred cheap addresses never touches a per-IP limit at all, because no single address is doing much.</p>



<p class="wp-block-paragraph">So layer keys instead of tuning one harder:</p>



<pre class="wp-block-code"><code>limit_req_zone $binary_remote_addr zone=perip:20m    rate=10r/s;
limit_req_zone $server_name        zone=persite:1m   rate=200r/s;</code></pre>



<p class="wp-block-paragraph">Applied together, the per-IP zone catches the loud individual and the per-site zone caps total damage regardless of how many addresses are involved. When several <code>limit_req</code> directives match, all of them are evaluated and the most restrictive result wins. If any one of them rejects, the request is rejected.</p>



<p class="wp-block-paragraph">One inheritance rule catches people out: <code>limit_req</code> directives are inherited from the enclosing level only if the current level defines none. Add a single <code>limit_req</code> inside a <code>location</code> and you have dropped every limit set at <code>server</code> level for that location. If you want both, restate both.</p>



<h3 class="wp-block-heading">Allowlisting with an empty key</h3>



<p class="wp-block-paragraph">Requests whose key evaluates to an empty string are not accounted at all. That is the clean way to exempt monitoring probes, your office, or a payment webhook source without maintaining a parallel set of locations.</p>



<pre class="wp-block-code"><code>geo $limit {
    default        1;
    10.0.0.0/8     0;
    192.0.2.10/32  0;
}

map $limit $limit_key {
    0 "";                    # empty key, never limited
    1 $binary_remote_addr;
}

limit_req_zone $limit_key zone=perip:20m rate=10r/s;</code></pre>



<p class="wp-block-paragraph">Nginx evaluates map variables only when they are used, so declaring a large map costs nothing on requests that never reference it.</p>



<h2 class="wp-block-heading">Roll it out in dry run, not in production</h2>



<p class="wp-block-paragraph">Both modules have a dry run mode. Excessive requests are counted in the shared memory zone and logged exactly as they would be, but nothing is actually rejected or delayed. Combined with the status variables, this gives you a real answer to &#8220;would this limit have hurt anyone&#8221; before you find out from a customer.</p>



<pre class="wp-block-code"><code>log_format ratelimit '$remote_addr $status "$request" '
                     'req=$limit_req_status conn=$limit_conn_status '
                     'ua="$http_user_agent"';

server {
    limit_req_dry_run  on;
    limit_conn_dry_run on;

    access_log /var/log/nginx/ratelimit.log ratelimit;
}</code></pre>



<p class="wp-block-paragraph"><code>$limit_req_status</code> reports <code>PASSED</code>, <code>DELAYED</code>, <code>REJECTED</code>, <code>DELAYED_DRY_RUN</code> or <code>REJECTED_DRY_RUN</code>. <code>$limit_conn_status</code> reports <code>PASSED</code>, <code>REJECTED</code> or <code>REJECTED_DRY_RUN</code>. A numbered rollout that has never embarrassed me:</p>



<ol class="wp-block-list">
<li>Add the zones and the <code>limit_req</code> directives with dry run on, plus the log format above.</li>

<li>Run for a full traffic cycle. A week if you have weekly patterns, longer if you have a monthly billing spike.</li>

<li>Count the would-be rejections by user agent and by address. If real browsers appear, your rate or burst is wrong, not your users.</li>

<li>Turn dry run off on the least critical location first. Watch 429 rates alongside your normal error budget.</li>

<li>Only then apply it to login, checkout, or anything that costs you money when it breaks.</li>
</ol>



<p class="wp-block-paragraph">Shipping the rate limit log into Loki and graphing it in Grafana makes step three take minutes instead of an evening with <code>awk</code>. Any log platform with decent label cardinality handling works; the point is having rejection counts broken down by zone and user agent, over time, next to your latency graphs.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Bot mitigation in layers, cheapest first</h2>



<p class="wp-block-paragraph">Rate limiting treats all clients the same. Bot mitigation is about deciding that some clients deserve different treatment before the limiter ever sees them. The useful mental model is a set of layers ordered by cost and by how easy each is to defeat.</p>



<h3 class="wp-block-heading">Layer one: robots.txt, and being honest about what it does</h3>



<p class="wp-block-paragraph">robots.txt is a request, not a control. Crawlers that respect it will honour a <code>Disallow</code> on your search endpoint, your faceted filter URLs and your infinite calendar pages, and that alone removes a surprising amount of load. Crawlers that do not respect it will ignore the file completely, and some will read it specifically to find out where the interesting URLs are.</p>



<p class="wp-block-paragraph">Use it to shape well-behaved traffic. Never use it as a security boundary. And be careful about blanket disallows on search engines you actually want: deindexing yourself is a much more expensive mistake than a bit of crawler load.</p>



<h3 class="wp-block-heading">Layer two: user agent classification with map</h3>



<p class="wp-block-paragraph">A <code>map</code> on <code>$http_user_agent</code> is cheap, readable, and easy to revise. It is also trivially spoofable, so treat it as traffic shaping rather than enforcement.</p>



<pre class="wp-block-code"><code>map $http_user_agent $ua_class {
    default          human;
    ""               empty;
    "~*bot|crawler|spider|scrape"  bot;
}

map $ua_class $bot_limit_key {
    bot   $binary_remote_addr;
    empty $binary_remote_addr;
    human "";
}

limit_req_zone $bot_limit_key zone=bots:10m rate=30r/m;</code></pre>



<p class="wp-block-paragraph">That gives declared bots a much tighter budget than browsers while leaving human traffic on the general limit. An empty user agent is worth its own class: almost nothing legitimate sends one, and it is a reliable signal for cheap scraping tools.</p>



<p class="wp-block-paragraph">Two cautions. Broad substring matching on <code>bot</code> will catch strings you did not intend, so review a sample of your own logs before deploying the pattern. And blocking rather than throttling is a bigger decision than it looks: if you return 403 to a crawler you actually want, you find out weeks later in your search traffic, not in your error log.</p>



<h3 class="wp-block-heading">Layer three: verifying bots that claim a name</h3>



<p class="wp-block-paragraph">The interesting case is a request claiming to be a major search crawler. Impersonating one is the standard way to bypass a user agent based allowlist, so a claim needs verification.</p>



<p class="wp-block-paragraph">The method the search vendors themselves document is a forward-confirmed reverse DNS check: reverse-resolve the client address, confirm the resulting hostname belongs to the vendor&#8217;s domain, then forward-resolve that hostname and confirm it returns the original address. An attacker cannot fake this without controlling the vendor&#8217;s DNS. Several vendors also publish their crawler IP ranges as machine-readable lists, which avoids the DNS round trip entirely.</p>



<p class="wp-block-paragraph">Here is the honest limitation: stock Nginx has no directive that does this. There is no built-in reverse DNS verification in the core modules. Your realistic options are a third-party module, doing the verification out of band and feeding the result into a <code>geo</code> block or an included allowlist file, or pushing the decision up to a CDN or WAF layer that already does it. Anyone showing you a pure stock-Nginx config that &#8220;verifies Googlebot&#8221; is verifying a string, not an identity.</p>



<p class="wp-block-paragraph">This is also the point where an edge provider earns its keep. Cloudflare, Fastly and similar edge platforms maintain verified bot lists and fingerprinting that no single origin server can reproduce, and they drop the traffic before it costs you a worker. If your bot problem is large and distributed, the edge is the right layer and Nginx is your backstop, not your front line.</p>



<h3 class="wp-block-heading">Layer four: escalation to the firewall</h3>



<p class="wp-block-paragraph">A client that collects hundreds of 429s and keeps going is not going to stop. Serving it a rejection still costs you a TCP handshake, a TLS negotiation and a log line. At that point the cheapest response is to stop answering at all.</p>



<p class="wp-block-paragraph">Fail2ban is the usual tool: a filter that matches your rate limit rejections in the Nginx error or access log, an action that inserts a firewall drop, and a ban time long enough to be inconvenient. Two things to get right. Match on a pattern that only fires for real limit rejections, or you will ban yourself during a deploy. And if you are behind a CDN, your firewall sees the CDN address, not the client, so the ban has to be applied at the edge instead. Banning a Cloudflare range at your origin firewall takes your whole site offline, and it is a mistake people make exactly once.</p>



<p class="wp-block-paragraph">Return code <code>444</code> is worth knowing here. It is Nginx-specific and closes the connection without any response, which is cheaper than generating an error page for a client that will never read it.</p>



<h2 class="wp-block-heading">Troubleshooting Nginx rate limiting</h2>



<p class="wp-block-paragraph"><strong>The limit does nothing.</strong> Check the key first. Log <code>$binary_remote_addr</code> alongside <code>$realip_remote_addr</code> and confirm they differ in the way you expect. Then check inheritance: a <code>limit_req</code> in the matched <code>location</code> discards the <code>server</code> level ones. Confirm which location actually matched by adding <code>$request_uri</code> and a marker header while testing.</p>



<p class="wp-block-paragraph"><strong>Real users are getting 429 or 503.</strong> Turn dry run back on and read the log rather than guessing. Usually it is one of three things: a burst too small for a normal page load&#8217;s asset count, a <code>limit_conn</code> value set for HTTP/1.1 assumptions on an HTTP/2 server, or a shared-egress network where many people genuinely share one address.</p>



<p class="wp-block-paragraph"><strong>Nothing appears in the error log.</strong> The default log level for rate limit refusals is <code>error</code>, and delays are logged one level below whatever you set. If you set <code>limit_req_log_level warn</code>, delays go to <code>info</code>, which your log level may be filtering out entirely. Grep for the limiting messages directly:</p>



<pre class="wp-block-code"><code>grep 'limiting requests' /var/log/nginx/error.log | tail -50
grep 'limiting connections' /var/log/nginx/error.log | tail -50</code></pre>



<p class="wp-block-paragraph"><strong>Intermittent errors under heavy load with no matching rate.</strong> Suspect zone exhaustion. Enlarge the zone and see whether the pattern changes. High-cardinality keys and distributed crawls are the usual cause.</p>



<p class="wp-block-paragraph"><strong>Health checks and uptime probes are being throttled.</strong> Add them to the allowlist map. Monitoring hitting a rate limit produces alerts about an outage that is not happening, which is worse than no monitoring.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Trusting forwarded headers from every source, which turns a per-IP limit into no limit at all.</li>

<li>Leaving <code>limit_req_status</code> at the default 503, so throttling looks like an outage in your own monitoring.</li>

<li>Using <code>burst</code> without <code>nodelay</code> on an abuse path, and quietly filling worker slots with held connections.</li>

<li>Setting a <code>limit_conn</code> value based on HTTP/1.1 intuition on a server serving HTTP/2 or HTTP/3.</li>

<li>Adding one <code>limit_req</code> in a location and silently dropping every server-level limit for that path.</li>

<li>Blocking user agent substrings without checking a sample of real logs first.</li>

<li>Applying identical limits to reads and writes. A cacheable GET and a POST that writes to the database do not deserve the same budget.</li>

<li>Banning CDN address ranges at the origin firewall because the logs show them as the client.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Verify the key before tuning the rate. Every other decision depends on it being correct.</li>

<li>Always start in dry run and measure a full traffic cycle before enforcing.</li>

<li>Use separate zones per traffic class. Search, login, API, static and general traffic have different shapes and should not share a bucket.</li>

<li>Pair <code>limit_req</code> with <code>limit_conn</code>. The first caps arrival rate, the second caps occupancy, and neither substitutes for the other.</li>

<li>Layer a per-server zone above your per-IP zone so distributed traffic still hits a ceiling.</li>

<li>Allowlist with an empty key rather than duplicating location blocks.</li>

<li>Export <code>$limit_req_status</code> to your log platform and graph rejections next to latency and upstream errors.</li>

<li>Keep CDN trust ranges in a generated include file that is refreshed on a schedule and validated with <code>nginx -t</code> before reload.</li>

<li>Throttle before you block. Blocking failures are invisible until they show up in your search traffic.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Nginx rate limiting stop a DDoS attack?</h3>



<p class="wp-block-paragraph">It helps with application-layer floods that reach your server, because rejecting a request is far cheaper than processing it. It does nothing about volumetric attacks that saturate your network link, since that traffic never gets to Nginx to be limited. For that you need capacity in front of the origin: a CDN or scrubbing service. Treat rate limiting as protection against abuse and accidental floods, not as DDoS mitigation.</p>



<h3 class="wp-block-heading">What rate should I actually set?</h3>



<p class="wp-block-paragraph">There is no universal number, and any guide that gives you one has not seen your traffic. Measure instead: run dry run mode, look at your real per-IP request distribution, and set the rate above the busiest legitimate client with headroom. Then set a separate, much tighter zone for the specific endpoints that are expensive or attacked.</p>



<h3 class="wp-block-heading">Should I use 429 or 503 for rejected requests?</h3>



<p class="wp-block-paragraph">429 in nearly every case. It communicates that the client should slow down, well-behaved clients back off on it, and it keeps your 5xx metrics meaningful. The default is 503 purely for historical reasons.</p>



<h3 class="wp-block-heading">Can Nginx rate limit per hour or per day?</h3>



<p class="wp-block-paragraph">Not with the stock module. It accepts requests per second and requests per minute only, so the slowest native rate is one request per minute expressed as <code>1r/m</code>. Longer windows need a third-party module, a shared store such as Redis via a scripting module, or enforcement in the application itself. For something like a password reset limit, the application is usually the better place anyway, because it can key on the account rather than the address.</p>



<h3 class="wp-block-heading">Do rate limit zones work across multiple Nginx servers?</h3>



<p class="wp-block-paragraph">Not in the open source build. Each instance keeps its own shared memory zone, so a client spread across four load-balanced servers effectively gets four times the budget. The <code>sync</code> parameter on <code>limit_req_zone</code> synchronises state across a cluster, but it is part of the commercial subscription. Without it, either divide your intended rate by your instance count, ensure your balancer pins clients consistently, or move the limit to a shared store.</p>



<h3 class="wp-block-heading">Will blocking AI crawlers hurt my SEO?</h3>



<p class="wp-block-paragraph">Blocking training crawlers is separate from blocking search indexing crawlers, and the distinction is worth checking per vendor before you write a rule, because some vendors run several crawlers with different purposes under related names. The risk is collateral damage: a broad user agent pattern that also matches a search crawler will remove you from results, and you will not notice quickly. Verify your pattern against real log samples, and prefer throttling over outright blocking while you are unsure.</p>



<h3 class="wp-block-heading">Where should rate limiting live: the CDN, Nginx, or the application?</h3>



<p class="wp-block-paragraph">All three, doing different jobs. The CDN handles volume and known-bad reputation before it costs you anything. Nginx handles per-client shaping and protects your workers from anything that gets through. The application handles limits that need identity or business context, like per-account quotas, which Nginx cannot see. Skipping the Nginx layer because you have a CDN leaves you exposed the moment someone finds your origin address.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Nginx rate limiting is not hard to configure. It is easy to configure into a state where it silently does nothing, or silently punishes the wrong people, and both failures look identical from the outside: a config that parses, a clean <code>nginx -t</code>, and no complaints until there are.</p>



<p class="wp-block-paragraph">So get the key right, prove it with a log line rather than assuming it, and run every new limit in dry run until the data tells you it is safe. Layer bot handling by cost, throttle before you block, and be honest about which layer can actually verify an identity. Everything else is tuning, and tuning is easy once the foundation is not lying to you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Nginx edge?</h2>



<p class="wp-block-paragraph">Most of the work here is diagnostic rather than clever. Things I regularly help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing <code>limit_req</code> and <code>limit_conn</code> setup to find out whether it is actually limiting anything, including the keying and inheritance traps above.</li>

<li>Fixing real client IP handling behind Cloudflare or another CDN, with a generated trust list that refreshes safely instead of drifting.</li>

<li>Designing per-endpoint zones for WordPress, WooCommerce or API origins so login, search and write paths get budgets that match their real cost.</li>

<li>Running a measured dry run rollout and reporting back what would have been rejected, broken down by user agent and network, before anything is enforced.</li>

<li>Building bot classification and escalation: user agent maps, allowlists, Fail2ban integration, and knowing which decisions belong at the edge instead.</li>

<li>Wiring rate limit telemetry into Grafana and Loki so throttling is something you can see on a dashboard rather than discover from a ticket.</li>
</ul>



<p class="wp-block-paragraph">If you would like a look at yours, send me the relevant server block, a sample of the access log, and whatever is currently going wrong. That is usually enough to say something useful straight away.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/nginx-rate-limiting-bot-mitigation/">Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>DirectAdmin PHP-FPM Pools: Why One WordPress Site Takes Down the Others</title>
		<link>https://john-nessime.com/blog/system-administration/directadmin-php-fpm-pools-wordpress-tuning/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 23 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[CustomBuild]]></category>
		<category><![CDATA[DirectAdmin]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OPcache]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[PHP-FPM]]></category>
		<category><![CDATA[Shared Hosting]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Website Performance]]></category>
		<category><![CDATA[WordPress Hosting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=246</guid>

					<description><![CDATA[<p>DirectAdmin gives each user one PHP-FPM pool, not each site. That single design fact explains most of the 502s on a busy shared box. Here is how to find the pool files, size pm.max_children from real memory numbers, set per-site limits that WordPress cannot override, and make the changes survive the next rewrite_confs.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/directadmin-php-fpm-pools-wordpress-tuning/">DirectAdmin PHP-FPM Pools: Why One WordPress Site Takes Down the Others</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;the shop is throwing 502s again.&#8221; The shop was fine. What was not fine was the half-forgotten news site sitting on the same DirectAdmin account, running a broken cron that fired an uncached admin-ajax call every few seconds. It had eaten every worker in the pool, and WooCommerce was queued behind it, waiting for a process that was never going to free up in time.</p>



<p class="wp-block-paragraph">That is the thing about <strong>DirectAdmin PHP-FPM pools</strong> that catches people out. You think you are tuning a site. You are not. You are tuning a user, and every domain that user owns is sharing the same worker budget, the same queue, and the same memory ceiling.</p>



<p class="wp-block-paragraph">This post covers where DirectAdmin actually keeps the pool config, how to size <code>pm.max_children</code> from numbers your server gives you rather than from a blog post&#8217;s magic value, which per-site PHP limits are worth setting and which are theatre, why OPcache does not behave the way you expect on a multi-tenant box, and how to make all of it survive the next CustomBuild run.</p>



<h2 class="wp-block-heading">The design fact that explains most of your 502s</h2>



<p class="wp-block-paragraph">DirectAdmin generates one FPM pool per user account, per PHP version. The pool is named after the Linux user. It listens on a Unix socket named after the user. It runs as that user. Apache or Nginx hands PHP requests to that socket for every domain and subdomain in the account.</p>



<p class="wp-block-paragraph">So if a reseller client has twelve domains under one username, those twelve sites share a single value of <code>pm.max_children</code>. There is no per-domain pool. This is a long-standing feature request rather than an oversight, and it has a real consequence: <strong>the noisiest site in an account sets the availability of every other site in that account.</strong></p>



<p class="wp-block-paragraph">Once you have internalised that, the fix for a lot of &#8220;mystery 502&#8221; tickets stops being a config tweak and starts being an account layout decision. If a site matters commercially, give it its own DirectAdmin user. That is the cleanest isolation the panel offers without going off-piste, and it costs you nothing but a bit of admin tidiness.</p>



<p class="wp-block-paragraph">The alternative, if you have User jails enabled, is DirectAdmin&#8217;s isolated FPM mode. That runs a separate <code>php-fpm</code> master process per jailed user instead of a shared master with many pools:</p>



<pre class="wp-block-code"><code>da config-set isolated_fpm 1
systemctl restart directadmin
da taskq --run 'action=rewrite&amp;value=httpd'</code></pre>



<p class="wp-block-paragraph">It only applies to users who already have Jailed enabled. The trade-off is memory: one master process per user, plus a separate OPcache region for each, adds up quickly. DirectAdmin mitigates that by letting idle per-user FPM instances exit after a period of inactivity, but on a box with sixty accounts you will feel it. I would reach for this on a server hosting a handful of paying clients, not on a general shared box.</p>



<h2 class="wp-block-heading">Where the pool config actually lives</h2>



<p class="wp-block-paragraph">This trips up almost everyone the first time, because there are two files with nearly the same name and only one of them matters for user sites.</p>



<ul class="wp-block-list">

<li><code>/usr/local/php83/etc/php-fpm.conf</code> is the master config for that PHP version. Editing it by hand is how you learn that CustomBuild overwrites it.</li>


<li><code>/usr/local/directadmin/data/users/USERNAME/php/php-fpm83.conf</code> is the generated per-user pool. This is the file that governs a customer&#8217;s sites.</li>


<li><code>/usr/local/directadmin/data/templates/php-fpm.conf</code> is the template those per-user files are rendered from.</li>

</ul>



<p class="wp-block-paragraph">Substitute your PHP version for <code>83</code> throughout. To see what every account is currently running:</p>



<pre class="wp-block-code"><code>ls /usr/local/directadmin/data/users/*/php/php-fpm*.conf

grep -H 'pm.max_children' /usr/local/directadmin/data/users/*/php/php-fpm*.conf</code></pre>



<p class="wp-block-paragraph">The second command is the one I run first on any server I have just inherited. It gives you the entire worker budget of the box on one screen, and it is usually the moment you discover that forty accounts are all sitting on the stock default.</p>



<p class="wp-block-paragraph">The DirectAdmin template ships with <code>pm = ondemand</code> and a low default child count, driven by a <code>MAX_CHILDREN</code> token. Ondemand means no workers exist until a request arrives, and idle workers are reaped after a timeout. On a shared box with many mostly-idle accounts, that is the right default and I would leave it alone. On a server dedicated to one busy WordPress site, the spawn latency on the first request after an idle period is real and switching to <code>dynamic</code> is defensible. Most people who switch to <code>dynamic</code> on a shared server are trading memory they do not have for a latency win they will not notice.</p>



<h2 class="wp-block-heading">Sizing pm.max_children without guessing</h2>



<p class="wp-block-paragraph">Every tuning guide gives you a formula. The formula is fine. The input everyone gets wrong is worker memory, because they use a number from someone else&#8217;s server instead of measuring their own. A lean WordPress install and the same install with a page builder, a security plugin and WooCommerce are not remotely the same workload.</p>



<p class="wp-block-paragraph">Measure it. This averages the resident memory of the workers currently running in one pool:</p>



<pre class="wp-block-code"><code>ps -eo rss,args | grep '&#91;p]hp-fpm: pool wpuser' 
  | awk '{sum+=$1; n++} END {if (n) printf "%d workers, avg %.0f MBn", n, sum/n/1024}'</code></pre>



<p class="wp-block-paragraph">The bracket trick in the grep pattern stops grep matching its own process. <code>rss</code> is resident set size in kilobytes, which is why the awk divides twice. Run it during a busy period, not at three in the morning, and run it a few times.</p>



<p class="wp-block-paragraph">Two honest caveats about RSS. It overstates real consumption, because workers forked from the same master share a lot of pages, including the OPcache region. And it understates your worst case, because a worker that has not yet hit a memory-hungry request looks cheap. Treat the average as a planning figure, not a guarantee, and leave headroom.</p>



<p class="wp-block-paragraph">Then the arithmetic:</p>



<ol class="wp-block-list">

<li>Take total RAM and subtract what MySQL or MariaDB, the web server, mail, and the panel itself need. On a WordPress box the database is usually the second-largest consumer and it is not optional.</li>


<li>Reserve a further slice for page cache and burst. I aim to leave a comfortable margin free rather than allocating to the last megabyte.</li>


<li>Divide what is left by your measured per-worker figure. That is your ceiling for <em>all</em> pools combined.</li>


<li>Split that ceiling across accounts by actual traffic, not evenly.</li>

</ol>



<p class="wp-block-paragraph">Step three is where people go wrong on cheap VPS plans. If you are running a dozen WordPress sites on one of the low-RAM tiers from a provider like Contabo or InterServer, the honest answer is often that you cannot give every account a generous pool and you need to either cut the site count or move up a tier. Setting <code>pm.max_children</code> to a number your RAM cannot back does not buy capacity. It buys the OOM killer, and the OOM killer usually takes the database with it, which turns a slow site into a broken one.</p>



<p class="wp-block-paragraph">To raise the server-wide default:</p>



<pre class="wp-block-code"><code>da config-set php_fpm_max_children_default 30
systemctl restart directadmin
da build rewrite_confs
systemctl restart php-fpm83</code></pre>



<p class="wp-block-paragraph">For a single account, go to <em>Admin Level &rarr; Custom HTTPD Configurations &rarr; domain.com</em> and open the <code>php-fpm</code> entry for that PHP version. In the <code>|CUSTOM1|</code> textarea, set the token:</p>



<pre class="wp-block-code"><code>|?MAX_CHILDREN=40|</code></pre>



<p class="wp-block-paragraph">That syntax is a template token override, not a config line. It tells the template renderer to use 40 instead of the default when it writes <code>pm.max_children</code> for that user. Writing <code>pm.max_children = 40</code> in that box instead is the single most common mistake here, and it does something worse than nothing: the template still emits its own <code>pm.max_children</code>, you now have the directive twice in one pool, and FPM refuses to start. On a server where one broken pool file stops the whole master, that takes down every site on the box.</p>



<p class="wp-block-paragraph">Rewrite and restart after any change, then confirm the value actually landed in the generated file rather than trusting the panel.</p>



<h2 class="wp-block-heading">Per-site PHP limits, and the difference that matters</h2>



<p class="wp-block-paragraph">The <code>|CUSTOM2|</code> textarea on the same page is where PHP settings go. The syntax is FPM pool syntax, not php.ini syntax:</p>



<pre class="wp-block-code"><code>php_admin_value&#91;memory_limit] = 512M
php_admin_value&#91;max_execution_time] = 120
php_value&#91;upload_max_filesize] = 64M
php_value&#91;post_max_size] = 64M
php_value&#91;max_input_vars] = 5000</code></pre>



<p class="wp-block-paragraph">The <code>admin</code> in <code>php_admin_value</code> is not decoration. Settings written with <code>php_value</code> can be raised at runtime by the application through <code>ini_set()</code>. Settings written with <code>php_admin_value</code> cannot. That distinction is the whole game on a multi-tenant server.</p>



<p class="wp-block-paragraph">WordPress and a good number of plugins will happily try to raise <code>memory_limit</code> at runtime, and some optimisation plugins push it to values that would let one bad request consume a large slice of the machine. If you want a genuine ceiling, use <code>php_admin_value</code>. If you want a sensible starting point that a developer can raise for a legitimate import job, use <code>php_value</code>. I use admin for memory and execution time, plain for upload sizes, because upload sizes are a support-ticket generator and not a stability risk.</p>



<p class="wp-block-paragraph"><code>max_input_vars</code> deserves its own note. It is the one that silently truncates a large WooCommerce product save or a menu with hundreds of items, with no error anywhere. The user reports &#8220;my changes did not save&#8221; and every log is clean. If you support content-heavy sites, raise it before someone asks.</p>



<h3 class="wp-block-heading">Letting site owners set their own</h3>



<p class="wp-block-paragraph">Under FPM there is no <code>.htaccess</code> route for PHP settings, because FPM is a separate process and never sees the file. DirectAdmin enables the per-directory alternative by default: a <code>.user.ini</code> file in <code>public_html</code>, controlled by the <code>user_ini.filename</code> setting. It handles the <code>PHP_INI_PERDIR</code> and <code>PHP_INI_USER</code> class of settings only, so it will never override anything you set with <code>php_admin_value</code>. That is the point of it.</p>



<h3 class="wp-block-heading">The WP-CLI gap</h3>



<p class="wp-block-paragraph">None of this applies to WP-CLI. Cron jobs and shell commands run through the CLI SAPI and read the CLI php.ini, not your pool. So a plugin update that works fine in the browser can die on memory in a cron-driven <code>wp cron event run</code>, and you will hunt through the pool config finding nothing wrong. Check the CLI limits separately when a scheduled task fails but the same action succeeds in wp-admin.</p>



<h2 class="wp-block-heading">OPcache is shared, and that is the invisible one</h2>



<p class="wp-block-paragraph">OPcache allocates its shared memory region when the FPM master starts, before any pool exists. Every pool under that master inherits the same region. You cannot meaningfully give one account its own OPcache budget through a pool directive, because by the time the pool is read the memory has already been carved out.</p>



<p class="wp-block-paragraph">Two consequences, and the second is the one nobody plans for.</p>



<p class="wp-block-paragraph">First, capacity. The default OPcache size is comfortable for one application and thin for thirty WordPress installs, each with its own core, theme and plugin tree. When it fills, OPcache starts evicting and eventually restarts, and every site on the version gets slower at once with nothing in any error log to explain it. Watch the wasted-memory and restart counters rather than guessing.</p>



<p class="wp-block-paragraph">Second, the sizing you actually need scales with the <em>total</em> number of PHP files across all accounts on that PHP version, not the biggest site. Consolidating everyone onto one PHP version looks tidy and quietly makes this worse. Spreading accounts across two or three PHP versions gives each master its own region, which is an underrated side effect of not forcing everyone to upgrade on the same day.</p>



<p class="wp-block-paragraph">OPcache is configured through CustomBuild rather than the pool. Edit the customised copy under CustomBuild&#8217;s <code>custom</code> directory so your changes survive updates, then rebuild and restart the FPM master.</p>



<h2 class="wp-block-heading">Making changes survive the next rebuild</h2>



<p class="wp-block-paragraph">Anything you type directly into a generated file is temporary. The next <code>rewrite_confs</code>, PHP update or panel update will flatten it, usually weeks later, and the regression will look like it came from nowhere.</p>



<p class="wp-block-paragraph">There are three durable places to put changes, in increasing order of blast radius:</p>



<ul class="wp-block-list">

<li><strong>Per user, all PHP versions.</strong> <code>/usr/local/directadmin/data/users/USERNAME/php/php-fpm.conf.custom1</code> and <code>.custom2</code>. These feed the same CUSTOM tokens as the panel textareas, so the settings follow the account when the user switches PHP version in Domain Setup. This is the one I reach for first.</li>


<li><strong>Server-wide, all users, all versions.</strong> <code>/usr/local/directadmin/data/templates/custom/php-fpm.conf.custom1</code> and <code>.custom2</code>. If either exists, its contents are injected into every pool file DirectAdmin writes. Good for a baseline you want everywhere.</li>


<li><strong>A full custom template.</strong> Copy <code>/usr/local/directadmin/data/templates/php-fpm.conf</code> to the <code>custom</code> subdirectory and edit it. This is the only route for pool directives that have no token, but you now own that file forever and stop receiving upstream template improvements. Use it when nothing else works, and leave a comment at the top saying why.</li>

</ul>



<p class="wp-block-paragraph">Global first, then per-user on top. Both land in the same token, so a per-user value does not replace a global one, it appends. For a directive that can only appear once in a pool, that is a duplicate and a failed restart.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;server reached max_children setting&#8221;</h3>



<p class="wp-block-paragraph">The pool name in that warning tells you which account, not which site. Find the log first rather than guessing at a path:</p>



<pre class="wp-block-code"><code>grep -i 'error_log' /usr/local/php83/etc/php-fpm.conf</code></pre>



<p class="wp-block-paragraph">Before raising the limit, check whether the requests are legitimate. Cross-reference the timestamps against each domain&#8217;s access log in that account. Nine times out of ten it is one of three things: an uncached <code>admin-ajax.php</code> loop from a plugin, a bot crawling faceted search or filtered archive URLs that bypass page cache, or <code>wp-cron.php</code> firing on every request on a site with a heavy scheduled task. Raising <code>pm.max_children</code> to absorb bot traffic is paying for garbage with RAM.</p>



<h3 class="wp-block-heading">502 or 504 with no PHP error</h3>



<p class="wp-block-paragraph">The request is dying at the FPM boundary rather than inside PHP. Usual causes are a worker killed by <code>request_terminate_timeout</code>, a full listen queue, or the master failing to start after a config change. Check the FPM service status before you check anything in WordPress. If FPM did not start at all, the panel will still show your settings saved and every site on that PHP version will be down.</p>



<h3 class="wp-block-heading">Changes in the panel that do nothing</h3>



<p class="wp-block-paragraph">Read the generated per-user file after every change. If the value is not in there, the rewrite did not run or the token was wrong. If it is in there and PHP still reports something else, the master was not restarted, or something later in the pool is overriding it. A temporary <code>phpinfo()</code> file served by the site in question settles the argument in seconds. Delete it immediately afterwards.</p>



<h3 class="wp-block-heading">Session or temp files that are not where you left them</h3>



<p class="wp-block-paragraph">If PHP writes to <code>/tmp</code> and you cannot see the files over SSH, that is systemd&#8217;s <code>PrivateTmp</code> giving the FPM service its own private namespace. Nothing is broken. The files exist, in a mount only that process can see.</p>



<h3 class="wp-block-heading">disable_functions that will not budge</h3>



<p class="wp-block-paragraph"><code>disable_functions</code> is special. From a pool you can add to the list but not remove from it, because the php.ini value has already been applied. To allow a function for one account you have to stop setting it in php.ini at all and drive the whole thing from templates instead. DirectAdmin documents that pattern, and it is worth reading before you start rather than after you have half-done it, because the intermediate state removes protection server-wide.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Editing <code>/usr/local/phpXX/etc/php-fpm.conf</code> directly and being surprised when it reverts.</li>


<li>Writing <code>pm.max_children = 40</code> into a CUSTOM box instead of the <code>|?MAX_CHILDREN=40|</code> token, producing a duplicate directive and a dead master.</li>


<li>Sizing workers from a memory figure found online rather than measured on the box.</li>


<li>Using <code>php_value</code> for <code>memory_limit</code> on shared hosting, then wondering how a plugin raised it to 2G.</li>


<li>Assuming pool settings apply to WP-CLI and cron.</li>


<li>Raising limits to absorb bot traffic instead of blocking or caching it upstream.</li>


<li>Putting a client&#8217;s revenue site in the same account as their abandoned projects.</li>

</ul>



<h2 class="wp-block-heading">Best practices for tuning DirectAdmin PHP-FPM pools</h2>



<ul class="wp-block-list">

<li>One DirectAdmin user per site that matters. Isolation you get for free beats isolation you have to engineer.</li>


<li>Keep <code>ondemand</code> on shared servers. Only move to <code>dynamic</code> when a single busy site owns the box.</li>


<li>Set a global baseline in the custom template, then override per user. Do not hand-tune forty files.</li>


<li>Use <code>php_admin_value</code> for anything that protects the server and <code>php_value</code> for anything that is only a convenience.</li>


<li>Cut PHP requests before you add workers. Full-page caching, a CDN in front, and Cloudflare or similar handling bots removes far more load than any pool setting.</li>


<li>Disable WordPress&#8217;s request-triggered cron and drive it from a real system cron, so scheduled work stops competing with visitors for workers.</li>


<li>Alert on saturation, not just uptime. Worker utilisation per pool and PHP-FPM error log entries are the signals worth having in Netdata, Prometheus, or whatever you already run. A site that is queueing looks perfectly healthy to an HTTP check.</li>


<li>Re-measure after major plugin changes. A page builder or a new security plugin moves per-worker memory enough to invalidate your arithmetic.</li>

</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I create a separate PHP-FPM pool for each domain in DirectAdmin?</h3>



<p class="wp-block-paragraph">Not through the panel. Pools are generated per user, per PHP version, and every domain in the account shares one. The supported ways to get separation are to move the domain to its own DirectAdmin user, or to enable isolated FPM mode for jailed users, which gives each user their own master process. Anything else means custom systemd units you maintain yourself, outside the panel&#8217;s knowledge.</p>



<h3 class="wp-block-heading">What is a good pm.max_children value for WordPress?</h3>



<p class="wp-block-paragraph">There is no portable number, because the input is your measured per-worker memory and your available RAM after the database. Measure the average resident size of your workers under real traffic, subtract everything else the server needs from total RAM, and divide. A lean brochure site and a WooCommerce store with thirty plugins can differ by a factor of three or more.</p>



<h3 class="wp-block-heading">Where does DirectAdmin store PHP-FPM pool configuration?</h3>



<p class="wp-block-paragraph">Generated per-user pools live at <code>/usr/local/directadmin/data/users/USERNAME/php/php-fpmXX.conf</code>, rendered from the template at <code>/usr/local/directadmin/data/templates/php-fpm.conf</code>. Do not edit either directly. Put durable changes in the custom files under the user&#8217;s <code>php</code> directory or in the <code>templates/custom</code> directory.</p>



<h3 class="wp-block-heading">Why do my PHP settings disappear after a CustomBuild update?</h3>



<p class="wp-block-paragraph">Because they were written into a generated file. CustomBuild and DirectAdmin regenerate those files on update and on any config rewrite. Only the custom template files and the CUSTOM token areas persist.</p>



<h3 class="wp-block-heading">Does .htaccess work for PHP settings under PHP-FPM?</h3>



<p class="wp-block-paragraph">No. FPM is a separate process from the web server and never reads <code>.htaccess</code>. Use a <code>.user.ini</code> file in the web root for the settings PHP allows at that level, or set the rest in the pool.</p>



<h3 class="wp-block-heading">Should I switch from ondemand to dynamic or static?</h3>



<p class="wp-block-paragraph">On a shared server, no. Ondemand exists precisely so that dozens of idle pools cost you nothing. On a server dedicated to one high-traffic site, <code>dynamic</code> removes cold-start latency and <code>static</code> gives the most predictable memory profile. The question to ask is whether your idle pools outnumber your busy ones.</p>



<h3 class="wp-block-heading">Can I give one WordPress site its own OPcache?</h3>



<p class="wp-block-paragraph">Not from the pool config. OPcache&#8217;s shared memory is allocated by the FPM master at startup and shared by every pool under it. The practical routes are isolated FPM mode, which gives jailed users their own master, or moving the site to a different PHP version so it sits under a different master.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">DirectAdmin PHP-FPM pools are a per-user boundary, not a per-site one. Every tuning decision that follows, how many workers to allow, whether a memory limit should be a suggestion or a wall, whether a client&#8217;s twelve domains belong in one account, comes out of that single fact.</p>



<p class="wp-block-paragraph">Measure your workers before you size them, use the CUSTOM token files so your work survives the next rebuild, and watch pool saturation rather than waiting for the 502 ticket. The site that goes down is rarely the site that caused it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a DirectAdmin server?</h2>



<p class="wp-block-paragraph">Most of the work I do on DirectAdmin boxes is this kind of thing: sites that are fine in isolation and fall over together. If that sounds familiar, I can help with:</p>



<ul class="wp-block-list">

<li>Auditing every pool on a server and producing a worker budget that matches the RAM you actually have</li>


<li>Tracking down which domain in a shared account is saturating the pool, and why</li>


<li>Building a global custom template so per-user tuning stops being forty hand-edited files</li>


<li>Splitting a multi-site DirectAdmin account into isolated users without breaking mail, DNS or SSL</li>


<li>OPcache and per-version sizing on servers hosting many WordPress installs</li>


<li>Wiring up pool-level monitoring and alerts so saturation shows up before the ticket does</li>

</ul>



<p class="wp-block-paragraph">Send me a pool file, the output of that RSS command, or the last hour of your PHP-FPM error log, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/directadmin-php-fpm-pools-wordpress-tuning/">DirectAdmin PHP-FPM Pools: Why One WordPress Site Takes Down the Others</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Troubleshooting High I/O Wait Without Blaming the Disk</title>
		<link>https://john-nessime.com/blog/devops/troubleshooting-high-io-wait-linux/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[bpftrace]]></category>
		<category><![CDATA[cgroups]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Disk I/O]]></category>
		<category><![CDATA[eBPF]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[iostat]]></category>
		<category><![CDATA[Kernel Tuning]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Performance Tuning]]></category>
		<category><![CDATA[Pressure Stall Information]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=248</guid>

					<description><![CDATA[<p>High I/O wait is a symptom, not a diagnosis. It rises when nothing is wrong and stays flat when something is badly wrong, and it only sees one layer of the storage stack. This is how to read iowait and PSI correctly, then work through the five failure families that actually produce storage stalls: device saturation, memory pressure, synchronous writes, throttling, and latency that lives above the block layer entirely.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/troubleshooting-high-io-wait-linux/">Troubleshooting High I/O Wait Without Blaming the Disk</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket says the app is slow. You SSH in, run <code>top</code>, and there it is: <code>wa</code> sitting at forty percent while user CPU barely moves. Obvious conclusion, the disk is dying. Somebody opens a ticket with the host, everyone waits, and the host replies that the volume looks healthy on their side.</p>



<p class="wp-block-paragraph">Sometimes the disk really is dying. More often it isn&#8217;t, and the number you&#8217;re staring at is not measuring what you think it&#8217;s measuring.</p>



<p class="wp-block-paragraph">This post covers troubleshooting high I/O wait the way I&#8217;d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.</p>



<h2 class="wp-block-heading">What high I/O wait actually measures</h2>



<p class="wp-block-paragraph">Here&#8217;s the definition that matters: iowait is a <em>subcategory of idle time</em>. For a given CPU, it&#8217;s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.</p>



<p class="wp-block-paragraph">Two consequences fall out of that, and both will burn you. <strong>High iowait can mean nothing is wrong.</strong> If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that&#8217;s a working system, not a fault.</p>



<p class="wp-block-paragraph"><strong>Low iowait can hide a severe problem.</strong> This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while <code>wa</code> reads two percent, because the CPU is never actually idle.</p>



<p class="wp-block-paragraph">Add the averaging problem on top. The <code>wa</code> figure in <code>top</code> is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press <code>1</code> in <code>top</code> to break out per-CPU lines before you conclude anything from the aggregate.</p>



<p class="wp-block-paragraph">So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.</p>



<h2 class="wp-block-heading">Start with pressure, not percentages</h2>



<p class="wp-block-paragraph">Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn&#8217;t get I/O?</p>



<pre class="wp-block-code"><code>cat /proc/pressure/io</code></pre>



<p class="wp-block-paragraph">You get two lines. <code>some</code> is the share of time at least one task was stalled waiting on I/O. <code>full</code> is the share of time <em>every</em> non-idle task was stalled at once, which is CPU capacity being burned for nothing. The <code>avg10</code>, <code>avg60</code> and <code>avg300</code> fields are percentages over the last 10, 60 and 300 seconds. <code>total</code> is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.</p>



<p class="wp-block-paragraph">The practical rule I use: a rising <code>full</code> line on <code>io</code> is a real user-visible problem almost every time. A high <code>some</code> line with a flat <code>full</code> line usually means one noisy job is waiting while everything else gets on with its life.</p>



<p class="wp-block-paragraph">If <code>/proc/pressure</code> doesn&#8217;t exist, the kernel was built without <code>CONFIG_PSI</code>, or it was built with PSI compiled in but disabled by default, in which case you need <code>psi=1</code> on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from &#8220;the host is stalling&#8221; to &#8220;this container is stalling&#8221; in one command:</p>



<pre class="wp-block-code"><code># Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;</code></pre>



<p class="wp-block-paragraph">If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are <code>node_pressure_io_waiting_seconds_total</code> for <code>some</code> and <code>node_pressure_io_stalled_seconds_total</code> for <code>full</code>. Rate those and you have a far better storage alert than any threshold on <code>%iowait</code> will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you&#8217;d rather not build the panels yourself.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the device really is saturated</h2>



<p class="wp-block-paragraph">This is the case everyone assumes, so get it confirmed or eliminated quickly.</p>



<pre class="wp-block-code"><code>iostat -xy 1</code></pre>



<p class="wp-block-paragraph">The <code>-x</code> gives extended per-device stats. The <code>-y</code> matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.</p>



<p class="wp-block-paragraph">Now the columns worth your attention:</p>



<ul class="wp-block-list">
<li><strong>r_await and w_await</strong>: average milliseconds a read or write took, queue time included. This is the closest thing to &#8220;how slow does storage feel&#8221;. Split reads from writes, because they fail very differently.</li>
<li><strong>aqu-sz</strong>: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.</li>
<li><strong>r/s, w/s, rkB/s, wkB/s</strong>: the actual workload. Compare against what the device or volume class is rated for.</li>
<li><strong>%util</strong>: treat with suspicion, see below.</li>
</ul>



<p class="wp-block-paragraph"><strong>The %util trap.</strong> That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent <code>%util</code> reading is a common and expensive way to change nothing. Use <code>aqu-sz</code> and the await columns instead. Similarly, <code>svctm</code> was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.</p>



<p class="wp-block-paragraph">Once you know the device is busy, find the culprit. <code>pidstat -d 1</code> gives per-process read and write rates from the same sysstat package and needs nothing special enabled. <code>iotop -o</code> is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the <code>SWAPIN</code> and <code>IO&gt;</code> columns come back empty until you turn it on:</p>



<pre class="wp-block-code"><code>sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0</code></pre>



<p class="wp-block-paragraph">And check the boring things before the clever ones. <code>dmesg -T | grep -iE 'i/o error|reset|medium'</code> catches a failing drive or a flapping controller. <code>smartctl -a /dev/sda</code> catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.</p>



<h2 class="wp-block-heading">Failure family two: memory pressure in an I/O costume</h2>



<p class="wp-block-paragraph">This is the failure family I&#8217;d check second, and it&#8217;s invisible if you only look at storage.</p>



<p class="wp-block-paragraph">When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the &#8220;disk problem&#8221; disappears.</p>



<pre class="wp-block-code"><code># si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory</code></pre>



<p class="wp-block-paragraph">If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.</p>



<p class="wp-block-paragraph">There&#8217;s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by <code>vm.dirty_background_ratio</code> and <code>vm.dirty_ratio</code>. Cross the background threshold and flusher threads start writing. Cross the hard <code>vm.dirty_ratio</code> threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.</p>



<h2 class="wp-block-heading">Failure family three: small synchronous writes</h2>



<p class="wp-block-paragraph">Throughput can be trivial and the system still crawls. A workload doing thousands of tiny <code>fsync</code> calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.</p>



<p class="wp-block-paragraph">Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through <code>innodb_flush_log_at_trx_commit</code>; PostgreSQL through <code>synchronous_commit</code>. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don&#8217;t let it be a thing somebody changed once at 2am.</p>



<p class="wp-block-paragraph">The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.</p>



<h2 class="wp-block-heading">Failure family four: throttled, not saturated</h2>



<p class="wp-block-paragraph">A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.</p>



<ul class="wp-block-list">
<li><strong>Cloud volume limits.</strong> Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as <code>BurstBalance</code> on burstable volume types and <code>VolumeQueueLength</code>; other providers publish equivalents. The fix is a volume class change, not a sysctl.</li>
<li><strong>cgroup limits you set yourself.</strong> On cgroup v2, <code>io.max</code> caps a cgroup at a given bytes-per-second or IOPS figure, and <code>io.weight</code> shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container&#8217;s own metrics announces it.</li>
<li><strong>Neighbours on shared hardware.</strong> On oversubscribed VPS plans your storage is shared, and someone else&#8217;s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.</li>
</ul>



<pre class="wp-block-code"><code># Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-&lt;id&gt;.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-&lt;id&gt;.scope/io.max</code></pre>



<p class="wp-block-paragraph">One more signal while you&#8217;re here. In <code>top</code>, the <code>st</code> column is steal time, meaning the hypervisor gave your vCPU&#8217;s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.</p>



<h2 class="wp-block-heading">Failure family five: the latency isn&#8217;t in the block layer</h2>



<p class="wp-block-paragraph">Here&#8217;s the failure mode that wastes the most time, because every storage metric you check will look healthy.</p>



<p class="wp-block-paragraph"><code>iostat</code> measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that&#8217;s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.</p>



<p class="wp-block-paragraph">The diagnosis is the gap. Measure latency at the application&#8217;s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.</p>



<pre class="wp-block-code"><code># Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10</code></pre>



<p class="wp-block-paragraph">If <code>ext4slower</code> is full of hundred-millisecond operations and <code>biolatency</code> shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, <code>nfsiostat</code> splits out RTT from execution time, which separates network cost from server cost. Install the tooling with <code>bpfcc-tools</code> on Debian and Ubuntu, or <code>bcc-tools</code> on RHEL, AlmaLinux and Rocky.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A working order for troubleshooting high I/O wait</h2>



<p class="wp-block-paragraph">On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.</p>



<ol class="wp-block-list">
<li>Read <code>/proc/pressure/io</code>. If <code>full</code> is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.</li>
<li>Check <code>/proc/pressure/memory</code> at the same time. If it&#8217;s rising too, fix memory first.</li>
<li>Run <code>iostat -xy 1</code> for thirty seconds. Note <code>r_await</code>, <code>w_await</code> and <code>aqu-sz</code> per device. Ignore <code>%util</code>.</li>
<li>List processes stuck in uninterruptible sleep: <code>ps -eo state,pid,comm | awk '$1 ~ /^D/'</code>. That&#8217;s your blocked set, and it names the workload.</li>
<li>Attribute the traffic with <code>pidstat -d 1</code>. Confirm the process you suspect is the process actually doing the I/O.</li>
<li>If await is high, decide saturated versus throttled. Check cgroup <code>io.max</code>, check provider volume metrics, check steal time.</li>
<li>If await is low but the application is slow, jump to the layer gap. Run <code>biolatency</code> and <code>ext4slower</code> together and compare.</li>
<li>Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Reading iostat&#8217;s first report.</strong> It&#8217;s a since-boot average. Use <code>-y</code>, or discard the first block.</li>
<li><strong>Trusting %util on NVMe or cloud volumes.</strong> Parallel devices break the assumption it was built on.</li>
<li><strong>Alerting on a fixed %iowait threshold.</strong> It fires on healthy backups and stays silent during real stalls on busy hosts.</li>
<li><strong>Changing the I/O scheduler first.</strong> Going from <code>mq-deadline</code> to <code>none</code> on NVMe is sometimes right, but it&#8217;s a tuning step, not a diagnosis, and it fixes nothing if you&#8217;re throttled.</li>
<li><strong>Adding swap to fix swapping.</strong> More swap on a memory-starved box means more disk traffic, not less.</li>
<li><strong>Stopping at the block layer.</strong> If every storage metric looks clean and the app is still slow, you&#8217;re measuring at the wrong layer, not measuring a healthy system.</li>
<li><strong>Testing with the wrong benchmark.</strong> A sequential <code>dd</code> tells you nothing about a workload doing random 4K synchronous writes. Use <code>fio</code> and match the real access pattern.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Record PSI continuously.</strong> Storage incidents are short and bursty. Without history you&#8217;re guessing about something that already ended.</li>
<li><strong>Capture a baseline while things are healthy.</strong> Save an <code>iostat -xy</code> sample and a PSI reading from a normal day. &#8220;Is 8ms bad?&#8221; is unanswerable without one.</li>
<li><strong>Keep sar running.</strong> The sysstat package can archive metrics on a schedule, which means <code>sar -d</code> can show you what happened at 3am. Enable it before you need it.</li>
<li><strong>Separate hot write paths.</strong> Database data, database logs and application logs on separate volumes stops one from starving the others.</li>
<li><strong>Cap container logs.</strong> An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.</li>
<li><strong>Alert on latency and pressure, not utilisation.</strong> Users feel <code>w_await</code> and PSI <code>full</code>. Nobody has ever felt a percentage of device busy time.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is a normal or acceptable I/O wait percentage?</h3>



<p class="wp-block-paragraph">There isn&#8217;t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.</p>



<h3 class="wp-block-heading">Can high I/O wait be caused by something other than the disk?</h3>



<p class="wp-block-paragraph">Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.</p>



<h3 class="wp-block-heading">Why does iostat show 100% utilisation when the disk isn&#8217;t busy?</h3>



<p class="wp-block-paragraph">Because <code>%util</code> measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read <code>aqu-sz</code> and the await columns instead.</p>



<h3 class="wp-block-heading">How do I find which process is causing I/O wait?</h3>



<p class="wp-block-paragraph">Start with <code>pidstat -d 1</code>, which needs no special kernel settings. List processes in uninterruptible sleep with <code>ps -eo state,pid,comm</code> and filter for state D to see who is actually blocked. <code>iotop -o</code> gives a live view but needs <code>kernel.task_delayacct=1</code> on recent kernels for its percentage columns to populate.</p>



<h3 class="wp-block-heading">Does adding more RAM reduce I/O wait?</h3>



<p class="wp-block-paragraph">Often, yes, and it&#8217;s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If <code>vmstat</code> shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.</p>



<h3 class="wp-block-heading">Should I change the I/O scheduler to fix high I/O wait?</h3>



<p class="wp-block-paragraph">Only after you know the device is the bottleneck. Check the current setting with <code>cat /sys/block/sda/queue/scheduler</code>. On NVMe, <code>none</code> is a common and sensible default because the hardware already does the queuing. On spinning disks, <code>mq-deadline</code> or <code>bfq</code> give better latency behaviour. It&#8217;s a tuning knob, not a diagnostic, and it will not help if you&#8217;re being throttled.</p>



<h3 class="wp-block-heading">Why is my application slow when iostat says storage is fine?</h3>



<p class="wp-block-paragraph">Because <code>iostat</code> only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run <code>ext4slower</code> or <code>xfsslower</code> alongside <code>biolatency</code>: if the filesystem is slow and the device is fast, the gap between them is your problem.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">High I/O wait is a symptom that points in a general direction, not a diagnosis. It&#8217;s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.</p>



<p class="wp-block-paragraph">Read PSI to find out whether anyone is actually suffering. Read <code>r_await</code>, <code>w_await</code> and <code>aqu-sz</code> to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you&#8217;re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need someone to look at your storage stalls?</h2>



<p class="wp-block-paragraph">I work with teams on exactly this kind of problem, usually the ones that have already survived a round of &#8220;the host says the disk is fine&#8221;. Things I can help with:</p>



<ul class="wp-block-list">
<li>Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to</li>
<li>Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation</li>
<li>Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward</li>
<li>Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements</li>
<li>Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers</li>
<li>Benchmarking with <code>fio</code> against your real access pattern so capacity decisions rest on something other than a <code>dd</code> run</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got an <code>iostat</code> capture, a PSI reading or just a graph that doesn&#8217;t make sense, send it over and I&#8217;ll tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/troubleshooting-high-io-wait-linux/">Troubleshooting High I/O Wait Without Blaming the Disk</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</title>
		<link>https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Fargate]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Lightsail]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Managed Hosting]]></category>
		<category><![CDATA[NAT Gateway]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=262</guid>

					<description><![CDATA[<p>Choosing between Lightsail, EC2 and ECS for a small client workload is not a performance question. It is a question about which constraints you are accepting and how expensive they are to reverse. Profiles of all three with where each wins and loses, the cost levers that actually move the bill, a six-step decision procedure, and the arguments that fall apart on contact with a real project.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;site feels slow, can you take a look.&#8221; What it turned into was a migration. The client&#8217;s Lightsail instance had outgrown its bundle, the plan was to move to EC2, and Lightsail ships an export-to-EC2 feature that makes that sound like a button press.</p>



<p class="wp-block-paragraph">The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.</p>



<p class="wp-block-paragraph">That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.</p>



<p class="wp-block-paragraph">This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn&#8217;t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.</p>



<h2 class="wp-block-heading">Why the exit path matters more than the launch</h2>



<p class="wp-block-paragraph">All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.</p>



<p class="wp-block-paragraph">Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which is cheapest for a small site.&#8221; It is &#8220;how much of this stack is portable, and who operates it after handover.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Amazon Lightsail: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.</li>

<li>The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.</li>

<li>Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.</li>

<li>IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the <em>default</em> VPC in the same account and the same region. Not a custom VPC. Not another account.</li>

<li>Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.</li>

<li>The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.</li>

<li>No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.</li>
</ul>



<p class="wp-block-paragraph">Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail&#8217;s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.</p>



<h2 class="wp-block-heading">Amazon EC2: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.</li>

<li>Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.</li>

<li>Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.</li>

<li>Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.</li>

<li>Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.</li>

<li>Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.</li>

<li>You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.</li>
</ul>



<p class="wp-block-paragraph">The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.</p>



<h2 class="wp-block-heading">Amazon ECS: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.</p>



<ul class="wp-block-list">
<li><strong>ECS on Fargate</strong> bills per vCPU-hour and GB-hour of the resources you <em>declare in the task definition</em>, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.</li>

<li><strong>ECS on EC2</strong> means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.</li>
</ul>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.</li>

<li>Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.</li>

<li>Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.</li>

<li>Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.</li>

<li>If the app is already containerized and CI already builds images, you are most of the way there.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.</li>

<li>Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.</li>

<li>Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.</li>

<li>Handover is harder. A client&#8217;s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.</li>
</ul>



<p class="wp-block-paragraph">The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.</p>



<pre class="wp-block-code"><code># Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"</code></pre>



<p class="wp-block-paragraph">Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.</p>



<h2 class="wp-block-heading">Lightsail vs EC2 vs ECS on the lines that move the bill</h2>



<p class="wp-block-paragraph">Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.</p>



<ul class="wp-block-list">
<li><strong>Egress.</strong> Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.</li>

<li><strong>NAT gateway.</strong> Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.</li>

<li><strong>Load balancing.</strong> Lightsail&#8217;s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.</li>

<li><strong>Public IPv4.</strong> Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.</li>

<li><strong>Log ingestion.</strong> CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.</li>

<li><strong>Idle resources.</strong> Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.</li>
</ul>



<h2 class="wp-block-heading">A decision procedure for Lightsail vs EC2 vs ECS</h2>



<p class="wp-block-paragraph">Run these in order. The first one that gives a clear answer is the answer.</p>



<ol class="wp-block-list">
<li><strong>Does this workload need to talk privately to other AWS resources?</strong> If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.</li>

<li><strong>Who operates this in a year?</strong> If the answer is &#8220;the client&#8217;s office manager&#8221; or &#8220;nobody,&#8221; bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.</li>

<li><strong>Is the app already containerized, with CI that builds images?</strong> If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.</li>

<li><strong>Is the load steady or spiky?</strong> Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.</li>

<li><strong>How much egress?</strong> Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail&#8217;s bundled allowance is a real argument.</li>

<li><strong>What is the audit story?</strong> If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.</li>
</ol>



<p class="wp-block-paragraph">If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works <em>before</em> you need it. The commands are short:</p>



<pre class="wp-block-code"><code># Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records</code></pre>



<p class="wp-block-paragraph">Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.</p>



<p class="wp-block-paragraph">If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:</p>



<pre class="wp-block-code"><code>aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1</code></pre>



<p class="wp-block-paragraph">You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.</p>



<p class="wp-block-paragraph">Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;Lightsail is just EC2 with a nicer console&#8221;</h3>



<p class="wp-block-paragraph">Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.</p>



<h3 class="wp-block-heading">&#8220;Start on Lightsail, migrate to EC2 when you outgrow it&#8221;</h3>



<p class="wp-block-paragraph">Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.</p>



<h3 class="wp-block-heading">&#8220;Fargate is serverless, so it scales to zero and costs nothing when idle&#8221;</h3>



<p class="wp-block-paragraph">The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word &#8220;serverless&#8221; gives you.</p>



<h3 class="wp-block-heading">&#8220;Containers are better practice, so ECS is the professional choice&#8221;</h3>



<p class="wp-block-paragraph">Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.</p>



<h3 class="wp-block-heading">&#8220;Reserved Instances make EC2 cheaper, so always pick EC2&#8221;</h3>



<p class="wp-block-paragraph">Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client&#8217;s behalf is a conversation, not a default.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Lightsail cheaper than EC2 for a small website?</h3>



<p class="wp-block-paragraph">Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.</p>



<h3 class="wp-block-heading">Can a Lightsail instance connect to an RDS database?</h3>



<p class="wp-block-paragraph">Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the <em>default</em> VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.</p>



<h3 class="wp-block-heading">Does everything migrate when I export Lightsail to EC2?</h3>



<p class="wp-block-paragraph">No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel &amp; WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.</p>



<h3 class="wp-block-heading">Is ECS overkill for a single small container?</h3>



<p class="wp-block-paragraph">Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.</p>



<h3 class="wp-block-heading">Why is my Fargate bill higher than the vCPU-hour rate suggests?</h3>



<p class="wp-block-paragraph">Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.</p>



<h3 class="wp-block-heading">Should I use Lightsail container service instead of ECS?</h3>



<p class="wp-block-paragraph">Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.</p>



<p class="wp-block-paragraph">One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.</p>



<p class="wp-block-paragraph">Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion before you commit?</h2>



<p class="wp-block-paragraph">Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:</p>



<ul class="wp-block-list">
<li>Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses</li>

<li>Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency</li>

<li>Planning and executing Lightsail to EC2 migrations, including the parts that do not export</li>

<li>Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it</li>

<li>Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy</li>

<li>Writing the handover runbook so the next person can operate it without calling you</li>
</ul>



<p class="wp-block-paragraph">If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</title>
		<link>https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Ansible Collections]]></category>
		<category><![CDATA[Ansible Vault]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=204</guid>

					<description><![CDATA[<p>A single Ansible run can take a blank VPS to a hardened, reproducible server. The hard part is not the tasks, it is the order. Here is the ordering that keeps you logged in, the failures that pass green and bite two weeks later, and how to make the second run as safe as the first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.</p>



<p class="wp-block-paragraph">That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report <code>changed</code>, do nothing useful, and leave you believing the server is hardened when it is not.</p>



<p class="wp-block-paragraph">This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.</p>



<h2 class="wp-block-heading">What a one-run bootstrap actually has to survive</h2>



<p class="wp-block-paragraph">A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.</p>



<p class="wp-block-paragraph">Four things break it, and they break it in different ways:</p>



<ul class="wp-block-list">
<li><strong>The chicken-and-egg problem.</strong> Ansible needs Python on the target to run almost any module, and a minimal image may not have it.</li>

<li><strong>Lockout.</strong> You remove your own access before proving the replacement access works.</li>

<li><strong>Non-idempotent tasks.</strong> The first run is fine. The second run is where the damage happens.</li>

<li><strong>Missing collections.</strong> The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with <code>ansible-core</code>.</li>
</ul>



<p class="wp-block-paragraph">Everything below is organised around those four, because that is how they actually show up.</p>



<h2 class="wp-block-heading">The chicken-and-egg problem: no Python, no modules</h2>



<p class="wp-block-paragraph">Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.</p>



<p class="wp-block-paragraph">The escape hatch is <code>ansible.builtin.raw</code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with <code>gather_facts: false</code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.</p>



<pre class="wp-block-code"><code>- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update &amp;&amp; apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:</code></pre>



<p class="wp-block-paragraph">Two details worth understanding rather than copying. <code>failed_when: false</code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And <code>ansible.builtin.setup</code> is the fact-gathering module called explicitly, which means from that point on <code>ansible_distribution</code> and friends are available even though the play started with gathering disabled.</p>



<p class="wp-block-paragraph">If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.</p>



<h2 class="wp-block-heading">Failure family one: locking yourself out while the playbook says ok</h2>



<p class="wp-block-paragraph">This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.</p>



<h3 class="wp-block-heading">The order that survives</h3>



<p class="wp-block-paragraph">The rule is simple: <strong>nothing that removes access happens until something has proven the new access works.</strong> In practice:</p>



<ol class="wp-block-list">
<li>Connect as whatever the provider handed you, usually <code>root</code> or a sudo-capable default user.</li>

<li>Create the admin account, add it to the sudo or wheel group, install its authorized key.</li>

<li>Write the sudoers drop-in and validate it with <code>visudo -cf</code>.</li>

<li><strong>Prove you can log in as that account and escalate.</strong> This is the step everyone skips.</li>

<li>Install the firewall and allow SSH <em>before</em> enabling it.</li>

<li>Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.</li>

<li>Only now disable root login and password authentication.</li>
</ol>



<p class="wp-block-paragraph">Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:</p>



<pre class="wp-block-code"><code>- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"</code></pre>



<p class="wp-block-paragraph">Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.</p>



<h3 class="wp-block-heading">The socket activation trap</h3>



<p class="wp-block-paragraph">This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. <code>ssh.socket</code> owns the listening port, and <code>Port</code> in <code>sshd_config</code> is not what decides where the daemon listens.</p>



<p class="wp-block-paragraph">So your playbook writes <code>Port 2222</code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.</p>



<p class="wp-block-paragraph">The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:</p>



<pre class="wp-block-code"><code># Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'</code></pre>



<p class="wp-block-paragraph">If <code>ssh.socket</code> is enabled, the override needs an empty <code>ListenStream=</code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.</p>



<p class="wp-block-paragraph">Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.</p>



<h3 class="wp-block-heading">Firewall ordering</h3>



<p class="wp-block-paragraph">UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an <code>UNREACHABLE</code> that will not recover. Rules go in first, always:</p>



<pre class="wp-block-code"><code>- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled</code></pre>



<p class="wp-block-paragraph">Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.</p>



<h2 class="wp-block-heading">Failure family two: the second run is the dangerous one</h2>



<p class="wp-block-paragraph">A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are <code>shell</code> and <code>command</code> tasks with no guard.</p>



<p class="wp-block-paragraph">An unguarded <code>shell</code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.</p>



<p class="wp-block-paragraph">Guard them properly:</p>



<ul class="wp-block-list">
<li><code>creates:</code> or <code>removes:</code> makes the task skip when the target state already exists. Prefer this.</li>

<li><code>changed_when: false</code> for read-only commands so they stop lying about changing things.</li>

<li><code>when:</code> driven by a registered check, for anything conditional.</li>
</ul>



<p class="wp-block-paragraph">The other idempotency trap is <code>lineinfile</code> against <code>sshd_config</code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:</p>



<pre class="wp-block-code"><code>- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false</code></pre>



<p class="wp-block-paragraph">Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so <code>validate:</code> on the template task with <code>sshd -t -f %s</code> will fail on a temporary file that has no host keys and no context. Run <code>sshd -t</code> as its own task afterwards instead, and let the handler restart only if it passes.</p>



<p class="wp-block-paragraph">Two prerequisites for the drop-in approach. The base <code>sshd_config</code> needs an <code>Include /etc/ssh/sshd_config.d/*.conf</code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the <em>first</em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.</p>



<h2 class="wp-block-heading">Failure family three: the collections are not where you think</h2>



<p class="wp-block-paragraph"><code>ansible-core</code> ships the <code>ansible.builtin</code> namespace and essentially nothing else. The full <code>ansible</code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.</p>



<p class="wp-block-paragraph">Things people assume are built in and are not: <code>ufw</code> and <code>timezone</code> live in <code>community.general</code>. <code>authorized_key</code>, <code>sysctl</code> and <code>mount</code> live in <code>ansible.posix</code>. Declare them and pin them:</p>



<pre class="wp-block-code"><code># requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml</code></pre>



<p class="wp-block-paragraph">Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. <code>ansible-lint</code> flags this and it is worth listening to.</p>



<h2 class="wp-block-heading">Failure family four: secrets in the repository</h2>



<p class="wp-block-paragraph">Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.</p>



<p class="wp-block-paragraph">Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:</p>



<pre class="wp-block-code"><code>ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass</code></pre>



<p class="wp-block-paragraph">The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Putting it together: how to bootstrap a VPS with Ansible end to end</h2>



<p class="wp-block-paragraph">Structure matters less than order, but a shape that works well is a single <code>site.yml</code> calling roles in dependency order, with the risky access changes fenced off at the end.</p>



<pre class="wp-block-code"><code># inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22</code></pre>



<p class="wp-block-paragraph">Then the run itself, in this order:</p>



<ol class="wp-block-list">
<li><strong>Reachability.</strong> Python check, fact gathering, timezone, hostname, package cache refresh.</li>

<li><strong>Identity.</strong> Admin user, group membership, authorized key, sudoers drop-in with <code>visudo -cf</code> validation.</li>

<li><strong>Verification gate.</strong> The task shown earlier. Nothing destructive runs before this passes.</li>

<li><strong>Firewall.</strong> Rules first, policy second, enable third.</li>

<li><strong>Unattended updates.</strong> <code>unattended-upgrades</code> on Debian and Ubuntu, <code>dnf-automatic</code> with its systemd timer on RHEL-family hosts.</li>

<li><strong>Brute-force protection.</strong> fail2ban with your own <code>/etc/fail2ban/jail.local</code>, never edits to <code>jail.conf</code>.</li>

<li><strong>Workload.</strong> Docker, a web server, a monitoring agent, whatever the box is for.</li>

<li><strong>Access teardown.</strong> Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.</li>
</ol>



<p class="wp-block-paragraph">When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:</p>



<pre class="wp-block-code"><code>- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60</code></pre>



<p class="wp-block-paragraph"><code>meta: reset_connection</code> does not accept a <code>when</code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.</p>



<h2 class="wp-block-heading">Test it before you point it at anything real</h2>



<p class="wp-block-paragraph">The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.</p>



<pre class="wp-block-code"><code># catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff</code></pre>



<p class="wp-block-paragraph">One caveat on <code>--check</code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong><code>UNREACHABLE</code> on the very first task.</strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with <code>ssh-keygen -R</code> and re-add the correct one with <code>ssh-keyscan</code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.</li>

<li><strong><code>UNREACHABLE</code> immediately after a firewall task.</strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.</li>

<li><strong>&#8220;Missing sudo password&#8221;.</strong> The sudoers drop-in is not in effect or has bad syntax. Validate with <code>visudo -cf</code> on the file itself before trusting it.</li>

<li><strong>Module not found under an FQCN.</strong> The collection is not installed on the machine running the playbook. Confirm with <code>ansible-galaxy collection list</code>.</li>

<li><strong>New group membership ignored.</strong> Adding a user to <code>docker</code> mid-play does not affect the already-open connection. Insert <code>meta: reset_connection</code> after the group change.</li>

<li><strong>fail2ban installed but not banning.</strong> On images without rsyslog, <code>/var/log/auth.log</code> may not exist, so a jail configured to read it never starts. Check <code>fail2ban-client status</code> and switch the jail to the systemd backend if the log file is absent.</li>

<li><strong>Everything green, port unchanged.</strong> Socket activation. Check <code>systemctl is-enabled ssh.socket</code> and <code>ss -tlnp</code>.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling root login in the same play that creates the replacement user, with no verification step between them.</li>

<li>Enabling UFW before the SSH allow rule exists.</li>

<li>Assuming <code>Port</code> in <code>sshd_config</code> is authoritative on every distribution.</li>

<li>Unguarded <code>shell</code> tasks that notify handlers, causing service restarts on every run.</li>

<li>Editing <code>jail.conf</code> instead of creating <code>jail.local</code>, so the next package update silently reverts it.</li>

<li>Committing plaintext secrets because &#8220;it is only a bootstrap playbook&#8221;.</li>

<li>Testing on a host you cannot afford to lose, with no console access configured.</li>

<li>Relying on <code>--check</code> as a substitute for a real run on a disposable host.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.</li>

<li>Put a verification task before every irreversible one, and let the play fail there rather than after.</li>

<li>Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.</li>

<li>Validate configuration before restarting the service that reads it. <code>sshd -t</code>, <code>visudo -cf</code>, <code>nginx -t</code>.</li>

<li>Pin collections in <code>requirements.yml</code> and install them in CI, not just locally.</li>

<li>Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.</li>

<li>Reboot once during testing. Half of the silent failures only appear after a restart.</li>

<li>Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you really bootstrap a VPS with Ansible in a single playbook run?</h3>



<p class="wp-block-paragraph">Yes, including changing the SSH user and port mid-run. The mechanism is <code>set_fact</code> to update the connection variables, <code>meta: reset_connection</code> to drop the cached connection, and <code>wait_for_connection</code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.</p>



<h3 class="wp-block-heading">Should I use cloud-init or Ansible for initial server setup?</h3>



<p class="wp-block-paragraph">Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.</p>



<h3 class="wp-block-heading">Do I need Terraform or OpenTofu as well?</h3>



<p class="wp-block-paragraph">Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.</p>



<h3 class="wp-block-heading">Why does my playbook work the first time and fail the second?</h3>



<p class="wp-block-paragraph">Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.</p>



<h3 class="wp-block-heading">Is changing the SSH port worth doing?</h3>



<p class="wp-block-paragraph">It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.</p>



<h3 class="wp-block-heading">How do I recover if the playbook locks me out?</h3>



<p class="wp-block-paragraph">Through the provider&#8217;s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.</p>



<h3 class="wp-block-heading">Can this run against several servers at once?</h3>



<p class="wp-block-paragraph">Yes, but use <code>serial: 1</code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.</p>



<p class="wp-block-paragraph">Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a bootstrap playbook you can actually rerun?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:</p>



<ul class="wp-block-list">
<li>Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.</li>

<li>Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.</li>

<li>Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.</li>

<li>Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.</li>

<li>Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.</li>

<li>Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.</li>
</ul>



<p class="wp-block-paragraph">Send me the playbook, a recap from a second run, or the output of <code>ss -tlnp</code> from the host that is behaving oddly, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</title>
		<link>https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Automatic HTTPS]]></category>
		<category><![CDATA[Caddy]]></category>
		<category><![CDATA[Caddyfile]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[mod_php]]></category>
		<category><![CDATA[MPM]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reverse Proxy]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[Web Server]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=195</guid>

					<description><![CDATA[<p>Benchmarks pick the wrong winner. Here is how Nginx, Apache and Caddy actually differ in production: how each one fails under load, who owns certificate renewal, and where configuration lives when a developer needs to change it. Includes a decision procedure and the mistakes I see most.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A message landed in a channel I was in: &#8220;We&#8217;re standardising on Nginx.&#8221; No traffic profile. No mention of what the current setup was actually failing at. No list of the applications that would have to move. The decision got made because everybody knows Nginx is fast.</p>



<p class="wp-block-paragraph">That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.</p>



<p class="wp-block-paragraph">So this post compares <strong>Nginx vs Apache vs Caddy</strong> on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.</p>



<h2 class="wp-block-heading">Why the benchmark comparison picks the wrong winner</h2>



<p class="wp-block-paragraph">Almost every throughput comparison you&#8217;ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you&#8217;re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.</p>



<p class="wp-block-paragraph">The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn&#8217;t most projects.</p>



<p class="wp-block-paragraph">For everyone else, the useful question is not &#8220;which is fastest&#8221; but &#8220;which chores am I choosing to keep.&#8221; That&#8217;s what the rest of this is about.</p>



<h2 class="wp-block-heading">Failure family one: the process model decides how you fall over</h2>



<p class="wp-block-paragraph">This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.</p>



<h3 class="wp-block-heading">Apache: a hard worker ceiling, and a queue behind it</h3>



<p class="wp-block-paragraph">Apache&#8217;s behaviour depends entirely on which Multi-Processing Module is loaded. With <code>prefork</code>, you get one process per connection, capped by <code>MaxRequestWorkers</code>. When you hit that cap, new connections don&#8217;t get rejected. They sit in the kernel&#8217;s listen backlog waiting for a worker to free up.</p>



<p class="wp-block-paragraph">The symptom is the worst kind: the site &#8220;hangs.&#8221; Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.</p>



<p class="wp-block-paragraph">Check which MPM you&#8217;re actually running before you theorise about anything:</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"</code></pre>



<p class="wp-block-paragraph">The reason prefork survives is <code>mod_php</code>. It isn&#8217;t safe with Apache&#8217;s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over <code>mod_proxy_fcgi</code> instead, which frees you to run <code>mpm_event</code> and gets you HTTP/2 as a side effect, since <code>mod_http2</code> disables itself under prefork.</p>



<p class="wp-block-paragraph">If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.</p>



<h3 class="wp-block-heading">Nginx: cheap connections, different limits</h3>



<p class="wp-block-paragraph">Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.</p>



<p class="wp-block-paragraph">Its ceiling is <code>worker_processes</code> multiplied by <code>worker_connections</code>, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That&#8217;s a genuine operational advantage: the failure announces itself.</p>



<p class="wp-block-paragraph">The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.</p>



<h3 class="wp-block-heading">Caddy: goroutines, and a softer edge</h3>



<p class="wp-block-paragraph">Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don&#8217;t tune a worker count at all.</p>



<p class="wp-block-paragraph">The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean &#8220;workers exhausted&#8221; line, you get memory growth and garbage collection pressure. There&#8217;s no knob you turned wrong, which is pleasant right up until you need to explain what happened.</p>



<h2 class="wp-block-heading">Failure family two: who owns the certificate lifecycle</h2>



<p class="wp-block-paragraph">This is the one that actually causes outages, and it is the sharpest genuine difference between the three.</p>



<p class="wp-block-paragraph">With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.</p>



<p class="wp-block-paragraph">The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:</p>



<pre class="wp-block-code"><code># Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"</code></pre>



<p class="wp-block-paragraph">The distinction that matters: <code>--deploy-hook</code> runs only when a certificate was actually renewed, so you&#8217;re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.</p>



<p class="wp-block-paragraph">Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.</p>



<p class="wp-block-paragraph">It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy&#8217;s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an <code>ask</code> endpoint you host:</p>



<pre class="wp-block-code"><code>{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}</code></pre>



<p class="wp-block-paragraph">Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older <code>interval</code> and <code>burst</code> rate-limiting options are no longer recommended and should be removed if you still have them in a config.</p>



<p class="wp-block-paragraph">One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.</p>



<h2 class="wp-block-heading">Failure family three: where the configuration lives</h2>



<p class="wp-block-paragraph">This is the axis people underweight, and it&#8217;s usually the one that actually decides the answer.</p>



<p class="wp-block-paragraph">Apache supports <code>.htaccess</code>: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.</p>



<p class="wp-block-paragraph">The costs are real and they&#8217;re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for <code>.htaccess</code> files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.</p>



<p class="wp-block-paragraph">Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with <code>try_files</code> rather than per-directory rule files. If you own the box, that&#8217;s a feature: nothing changes routing behind your back and the running configuration is exactly what&#8217;s in version control. If you host other people&#8217;s applications, it&#8217;s a support burden, because every customer rewrite rule becomes a ticket for you.</p>



<p class="wp-block-paragraph">Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Nginx: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you&#8217;ll meet has a known-good Nginx config, and everyone you hire has already read one.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the DNS behaviour catches people constantly. A hostname written directly into <code>proxy_pass</code>, or into an <code>upstream</code> block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says &#8220;stale DNS&#8221;; you just get connection timeouts to an address that looks plausible.</p>



<p class="wp-block-paragraph">Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that&#8217;s what moves the lookup from config parse time to request time:</p>



<pre class="wp-block-code"><code>location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}</code></pre>



<p class="wp-block-paragraph">Note the <code>$request_uri</code> on the end. Once a variable appears in <code>proxy_pass</code>, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream&#8217;s root. Also worth knowing: an <code>upstream</code> block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.</p>



<p class="wp-block-paragraph">The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But &#8220;who decides what ships&#8221; is a fair question to ask about any dependency, and this one now has a public answer.</p>



<h2 class="wp-block-heading">Apache: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> memory per connection, if you&#8217;re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that&#8217;s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.</p>



<p class="wp-block-paragraph">I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and <code>mod_php</code> to <code>mpm_event</code> and PHP-FPM first, then measure, and quite often stop there.</p>



<h2 class="wp-block-heading">Caddy: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.</p>



<pre class="wp-block-code"><code>app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}</code></pre>



<p class="wp-block-paragraph">That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with <code>xcaddy</code> rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there&#8217;s a subtler cost: automation you don&#8217;t understand is still automation you&#8217;re responsible for. Being paged about ACME behaviour you&#8217;ve never had to learn is a worse afternoon than being paged about Certbot.</p>



<p class="wp-block-paragraph">On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the <code>dynamic a</code> upstream module with a <code>refresh</code> interval is the built-in answer, which is a cleaner story than Nginx&#8217;s variable trick.</p>



<h2 class="wp-block-heading">A decision procedure for Nginx vs Apache vs Caddy</h2>



<p class="wp-block-paragraph">Work down this list and stop at the first one that matches. It&#8217;ll be right more often than a benchmark.</p>



<ol class="wp-block-list">
<li><strong>Do applications you don&#8217;t control need to change server behaviour?</strong> If yes, Apache. Nothing else has <code>.htaccess</code>, and reimplementing it in tickets is not a plan.</li>

<li><strong>Is TLS renewal your top operational risk?</strong> Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.</li>

<li><strong>Are you terminating a very high connection rate, or serving heavy static content?</strong> Nginx. This is the workload it was designed around.</li>

<li><strong>Does the team already run one of these well?</strong> Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.</li>

<li><strong>Is there a working Apache install and no specific complaint?</strong> Fix the MPM and PHP handler, then reassess. Most &#8220;we need Nginx&#8221; tickets are actually &#8220;we need to stop using prefork.&#8221;</li>

<li><strong>None of the above?</strong> Nginx for anything that will be handed to another engineer. Caddy for anything you&#8217;ll run yourself.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Renewing certificates without reloading.</strong> The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.</li>

<li><strong>Migrating for benchmark numbers.</strong> You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.</li>

<li><strong>Leaving Apache on prefork after moving to PHP-FPM.</strong> The migration is only half done, and you&#8217;re still paying prefork&#8217;s memory cost while also giving up HTTP/2.</li>

<li><strong>Hostnames baked into <code>proxy_pass</code> with dynamic upstreams.</strong> Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.</li>

<li><strong>Enabling Caddy&#8217;s on-demand TLS without an <code>ask</code> endpoint.</strong> You&#8217;ve handed strangers the ability to make you request certificates.</li>

<li><strong>Exposing Caddy&#8217;s admin API beyond loopback.</strong> Unauthenticated full config replacement is exactly as bad as it sounds.</li>

<li><strong>Monitoring the certificate file instead of the handshake.</strong> The file is almost never the thing that&#8217;s wrong. Check what the server actually presents, from outside the box.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide, in practice</h2>



<ul class="wp-block-list">
<li>Pick based on who has to change the config six months from now, not on requests per second today.</li>

<li>Whichever you choose, dump the effective configuration rather than reading files: <code>nginx -T</code>, <code>apachectl -S</code>, <code>caddy adapt --config /etc/caddy/Caddyfile</code>. Include files hide surprises.</li>

<li>Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.</li>

<li>Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.</li>

<li>Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.</li>

<li>Validate before reloading, every time. <code>nginx -t</code>, <code>apachectl configtest</code> and <code>caddy validate</code> all exist so you don&#8217;t find out from users.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Caddy production ready?</h3>



<p class="wp-block-paragraph">Yes. It&#8217;s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.</p>



<h3 class="wp-block-heading">Can I run Nginx and Apache together?</h3>



<p class="wp-block-paragraph">Yes, and it&#8217;s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their <code>.htaccess</code> behaviour. You get Nginx&#8217;s connection handling and Apache&#8217;s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.</p>



<h3 class="wp-block-heading">Which is best for WordPress?</h3>



<p class="wp-block-paragraph">Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into <code>try_files</code> and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.</p>



<h3 class="wp-block-heading">Does Nginx really re-resolve DNS for upstreams?</h3>



<p class="wp-block-paragraph">Not by default. A literal hostname in <code>proxy_pass</code> or an <code>upstream</code> block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a <code>resolver</code> directive, and remembering that this changes how the request URI is passed.</p>



<h3 class="wp-block-heading">Should I switch from mod_php to PHP-FPM?</h3>



<p class="wp-block-paragraph">In almost all cases yes. It&#8217;s the change that unlocks Apache&#8217;s threaded MPMs and HTTP/2, and it separates PHP&#8217;s lifecycle from the web server&#8217;s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or <code>.htaccess</code>, since those move to a <code>.user.ini</code> file with different syntax.</p>



<h3 class="wp-block-heading">Does the freenginx fork mean I should stop using Nginx?</h3>



<p class="wp-block-paragraph">No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn&#8217;t put a client on a fork without a specific reason.</p>



<h3 class="wp-block-heading">Which uses the least memory?</h3>



<p class="wp-block-paragraph">Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on <code>mpm_event</code>, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Nginx vs Apache vs Caddy</strong> question is not a performance question. All three are fast enough for what you&#8217;re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.</p>



<p class="wp-block-paragraph">Pick the set of chores your team can actually carry. Then go fix the thing that&#8217;s genuinely slow, which is almost certainly not the web server.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your web server setup?</h2>



<p class="wp-block-paragraph">Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does</li>

<li>Migrating Apache from prefork and <code>mod_php</code> to <code>mpm_event</code> with PHP-FPM, without breaking existing <code>.htaccess</code> behaviour</li>

<li>Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches</li>

<li>Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring</li>

<li>Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted</li>

<li>Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me your config, an error log excerpt, or the output of <code>nginx -T</code> and tell me what you&#8217;re seeing. I&#8217;d rather look at the real thing than guess from a description.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Automating Server Provisioning for New Clients Without Building a Snowflake Farm</title>
		<link>https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Freelancing]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=192</guid>

					<description><![CDATA[<p>Hand-built client servers fail quietly: you only find out they were never reproducible on the day you need to rebuild one. Here is how to automate server provisioning for new clients using cloud-init for first boot, Ansible for convergence, per-client secrets and a verification gate that refuses to hand over a half-built box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email is usually short. &#8220;Can you spin up a staging box that matches production?&#8221; You open your notes from eight months ago, find a half-finished shell history and a text file called <code>setup-notes-final2.txt</code>, and the honest answer is no. You can build something close. You cannot build the same machine.</p>



<p class="wp-block-paragraph">That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between &#8220;the server works&#8221; and &#8220;the server is reproducible&#8221; turns into a weekend.</p>



<p class="wp-block-paragraph">This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client&#8217;s credentials from ever touching another client&#8217;s box, and how to prove a build is finished instead of assuming it.</p>



<h2 class="wp-block-heading">The four failure families worth designing against</h2>



<p class="wp-block-paragraph">Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.</p>



<h3 class="wp-block-heading">1. You cannot rebuild it</h3>



<p class="wp-block-paragraph">The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run <code>apt install</code>, every file edited over SSH at 1am, every &#8220;I&#8217;ll document that later&#8221; is a piece of the machine that exists nowhere but the machine.</p>



<p class="wp-block-paragraph">This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.</p>



<h3 class="wp-block-heading">2. Every client is a slightly different server</h3>



<p class="wp-block-paragraph">Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.</p>



<p class="wp-block-paragraph">Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts &#8220;all our clients have&#8230;&#8221; becomes a guess.</p>



<h3 class="wp-block-heading">3. One credential, too much blast radius</h3>



<p class="wp-block-paragraph">This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client&#8217;s server, and the same API token provisions every client&#8217;s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.</p>



<p class="wp-block-paragraph">Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.</p>



<h3 class="wp-block-heading">4. The handover you cannot perform</h3>



<p class="wp-block-paragraph">Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How to automate server provisioning for new clients</h2>



<p class="wp-block-paragraph">The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.</p>



<h3 class="wp-block-heading">Layer 1: the machine itself</h3>



<p class="wp-block-paragraph">Terraform or OpenTofu talks to your provider&#8217;s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client&#8217;s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.</p>



<p class="wp-block-paragraph">Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.</p>



<h3 class="wp-block-heading">Layer 2: first boot with cloud-init</h3>



<p class="wp-block-paragraph">cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.</p>



<pre class="wp-block-code"><code>#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo</code></pre>



<p class="wp-block-paragraph">Two deliberate omissions. There is no <code>package_upgrade: true</code>, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in <code>runcmd</code> is a script you will never test again.</p>



<p class="wp-block-paragraph"><code>python3</code> is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.</p>



<p class="wp-block-paragraph">Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:</p>



<pre class="wp-block-code"><code>cloud-init schema --config-file user-data.yaml</code></pre>



<h3 class="wp-block-heading">Layer 3: convergence with Ansible</h3>



<p class="wp-block-paragraph">This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.</p>



<p class="wp-block-paragraph">The layout that scales across clients separates the shared build from the per-client differences:</p>



<pre class="wp-block-code"><code>infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml</code></pre>



<p class="wp-block-paragraph">Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.</p>



<p class="wp-block-paragraph">Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:</p>



<pre class="wp-block-code"><code>cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  &gt; ../inventory/hosts.ini</code></pre>



<p class="wp-block-paragraph">There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.</p>



<h3 class="wp-block-heading">Layer 4: the verification gate</h3>



<p class="wp-block-paragraph">This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.</p>



<p class="wp-block-paragraph">Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:</p>



<pre class="wp-block-code"><code>- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost</code></pre>



<p class="wp-block-paragraph">Note <code>ssh_unit</code> as a variable rather than a literal: the SSH unit is <code>ssh.service</code> on Debian and Ubuntu but <code>sshd.service</code> on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.</p>



<p class="wp-block-paragraph">The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.</p>



<h2 class="wp-block-heading">Keeping clients apart: secrets and access</h2>



<p class="wp-block-paragraph">Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client&#8217;s infrastructure is a single, contained action.</p>



<p class="wp-block-paragraph">For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.</p>



<pre class="wp-block-code"><code># .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...</code></pre>



<p class="wp-block-paragraph">Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.</p>



<p class="wp-block-paragraph">For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.</p>



<p class="wp-block-paragraph">If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.</p>



<h2 class="wp-block-heading">Troubleshooting a provisioning run</h2>



<h3 class="wp-block-heading">Ansible connects before cloud-init has finished</h3>



<p class="wp-block-paragraph">The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.</p>



<p class="wp-block-paragraph">Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:</p>



<pre class="wp-block-code"><code>cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log</code></pre>



<p class="wp-block-paragraph">One caveat that catches people: never put <code>cloud-init status --wait</code> inside a <code>runcmd</code> or <code>bootcmd</code> block. It will deadlock waiting for the process that is running it.</p>



<h3 class="wp-block-heading">cloud-init appears to have done nothing</h3>



<p class="wp-block-paragraph">Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in <code>/var/lib/cloud/instance/sem/</code> are preventing a rerun.</p>



<pre class="wp-block-code"><code># what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot</code></pre>



<p class="wp-block-paragraph">Never run <code>cloud-init clean</code> on a production host. It removes cached state and the following boot will behave as a first boot.</p>



<h3 class="wp-block-heading">Host key verification failed on a rebuilt server</h3>



<p class="wp-block-paragraph">Rebuild a machine on the same IP and every client with the old key in <code>known_hosts</code> refuses to connect. The tempting fix is to disable host key checking in <code>ansible.cfg</code>. Do not. That turns a real integrity check into a permanent no-op across every client you manage.</p>



<p class="wp-block-paragraph">Remove the stale entry and pin the new one as part of the provisioning flow:</p>



<pre class="wp-block-code"><code>ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 &gt;&gt; ~/.ssh/known_hosts</code></pre>



<p class="wp-block-paragraph">Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.</p>



<h3 class="wp-block-heading">The playbook reports changes on every single run</h3>



<p class="wp-block-paragraph">Almost always a <code>shell</code> or <code>command</code> task with no <code>creates</code>, <code>changed_when</code> or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.</p>



<p class="wp-block-paragraph">Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:</p>



<pre class="wp-block-code"><code>ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Putting the application build in <code>runcmd</code>. It runs once, never gets tested, and is invisible to every later change.</li>

<li>One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.</li>

<li>Copying a role into a client directory to make one small change, instead of extracting a variable.</li>

<li>Treating &#8220;the playbook exited zero&#8221; as proof the server is ready. It is proof no task threw.</li>

<li>Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.</li>

<li>Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.</li>

<li>Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Rebuild before you need to.</strong> Destroy and rebuild a client&#8217;s staging host on a schedule. It is the only honest test of reproducibility.</li>

<li><strong>Version your roles and pin them per client.</strong> A shared role that everyone tracks on main means one commit changes six clients at once.</li>

<li><strong>Keep first boot minimal.</strong> If it can wait for Ansible, it should.</li>

<li><strong>Make monitoring part of provisioning.</strong> Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.</li>

<li><strong>Same for backups.</strong> An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.</li>

<li><strong>Separate credentials per client, always.</strong> Keys, tokens, encryption keys, state backends.</li>

<li><strong>Write the runbook as you build.</strong> The provisioning repo&#8217;s README is the handover document. If it is accurate, the engagement can end cleanly.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Terraform if I only manage a few VPS instances?</h3>



<p class="wp-block-paragraph">Not necessarily. If you are creating three servers a year through a provider&#8217;s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.</p>



<h3 class="wp-block-heading">Should I build custom images instead of provisioning at boot?</h3>



<p class="wp-block-paragraph">Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.</p>



<h3 class="wp-block-heading">Terraform or OpenTofu for client projects?</h3>



<p class="wp-block-paragraph">Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.</p>



<h3 class="wp-block-heading">How do I automate server provisioning without exposing SSH to the internet?</h3>



<p class="wp-block-paragraph">The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.</p>



<h3 class="wp-block-heading">How do I stop configuration drift once the server is live?</h3>



<p class="wp-block-paragraph">Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.</p>



<h3 class="wp-block-heading">What belongs in cloud-init versus Ansible?</h3>



<p class="wp-block-paragraph">cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.</p>



<h3 class="wp-block-heading">How long should provisioning a new client server take?</h3>



<p class="wp-block-paragraph">Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer &#8220;can you rebuild this?&#8221; with yes, on any day, without opening a shell on the existing machine.</p>



<p class="wp-block-paragraph">Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.</p>



<p class="wp-block-paragraph">Start with one client. Get their staging box rebuilding from scratch. Then make the second client&#8217;s build reuse the roles from the first. That is the whole path.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a provisioning setup you can actually hand over?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:</p>



<ul class="wp-block-list">
<li>Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime</li>

<li>Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets</li>

<li>Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works</li>

<li>Writing the verification playbook that decides whether a build is finished, and wiring it into CI</li>

<li>Fixing idempotency so check-mode runs become a usable drift report instead of noise</li>

<li>Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</title>
		<link>https://john-nessime.com/blog/devops/docker-log-rotation/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 10 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Container Runtime]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Disk Space]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[journald]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Logs]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsyslog]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Syslog]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=182</guid>

					<description><![CDATA[<p>Docker's default logging driver writes container output to a JSON file with no size limit and no rotation. This is a practical guide to Docker log rotation: what the defaults actually do, why your daemon.json change did nothing, why deleting the log file did not free any disk, and how to choose between json-file, local, journald and shipping logs off the box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-log-rotation/">Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Monitoring says the API is healthy. The ticket says nobody can deploy. You SSH in, run <code>df -h</code>, and the root filesystem is at 100%. A few minutes of poking around <code>/var/lib/docker</code> later you find it: one container, one file, a double-digit number of gigabytes of newline-delimited JSON, written continuously since the day that container was created.</p>



<p class="wp-block-paragraph">Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker&#8217;s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.</p>



<p class="wp-block-paragraph">This post covers Docker log rotation properly: what the defaults actually do, why the <code>daemon.json</code> change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.</p>



<h2 class="wp-block-heading">The default is unbounded, and that is documented behaviour</h2>



<p class="wp-block-paragraph">Docker&#8217;s default logging driver is <code>json-file</code>. It captures stdout and stderr from the container&#8217;s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.</p>



<p class="wp-block-paragraph">The important part is the option defaults. For <code>json-file</code>, <code>max-size</code> defaults to unlimited, <code>max-file</code> defaults to 1, and <code>compress</code> defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.</p>



<p class="wp-block-paragraph">Docker&#8217;s own documentation is upfront about why. Keeping <code>json-file</code> unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.</p>



<p class="wp-block-paragraph">Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:</p>



<pre class="wp-block-code"><code># What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"</code></pre>



<p class="wp-block-paragraph">The gap between the first command and the second is where most of the pain lives, and we will come back to it.</p>



<p class="wp-block-paragraph">To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container&#8217;s log file plus a small amount of metadata, so the sizes are close enough to be useful:</p>



<pre class="wp-block-code"><code>sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10</code></pre>



<p class="wp-block-paragraph">Container IDs are not memorable, so map the winner back to a name with <code>docker ps --no-trunc</code> or by grepping the ID against <code>docker ps -aq</code>. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.</p>



<h2 class="wp-block-heading">Setting up Docker log rotation on the daemon</h2>



<p class="wp-block-paragraph">The daemon-wide fix goes in <code>/etc/docker/daemon.json</code>. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.</p>



<pre class="wp-block-code"><code>{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}</code></pre>



<p class="wp-block-paragraph">What each key buys you:</p>



<ul class="wp-block-list">
<li><code>max-size</code> caps a single file. Accepts an integer plus <code>k</code>, <code>m</code> or <code>g</code>. This is the setting that stops the runaway.</li>

<li><code>max-file</code> caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is <em>only effective if <code>max-size</code> is also set</em>, which is the single most common misconfiguration I see.</li>

<li><code>compress</code> gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.</li>
</ul>



<p class="wp-block-paragraph">Two things trip people up here. First, every value in <code>log-opts</code> must be a JSON string, including the numeric ones. Writing <code>"max-file": 3</code> without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed <code>daemon.json</code> on a production host is a much worse afternoon than a full disk:</p>



<pre class="wp-block-code"><code># Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  &amp;&amp; echo "daemon.json parses OK"

sudo systemctl restart docker</code></pre>



<p class="wp-block-paragraph">Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is <code>max-size</code> multiplied by <code>max-file</code>. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.</p>



<h2 class="wp-block-heading">Why your daemon.json change did nothing</h2>



<p class="wp-block-paragraph">This is the failure mode that costs the most time, because it looks exactly like the fix not working.</p>



<p class="wp-block-paragraph">Logging configuration is baked into a container&#8217;s host config when the container is <em>created</em>. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because <code>docker restart</code> stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.</p>



<p class="wp-block-paragraph">So the container that filled your disk keeps filling your disk, its log file still has no cap, and <code>docker info</code> cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.</p>



<p class="wp-block-paragraph">To actually apply it, recreate the container and then verify rather than assuming:</p>



<pre class="wp-block-code"><code># Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web</code></pre>



<p class="wp-block-paragraph">You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.</p>



<p class="wp-block-paragraph">Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.</p>



<h3 class="wp-block-heading">Pinning limits per service in Compose</h3>



<p class="wp-block-paragraph">Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides <code>daemon.json</code> entirely. A YAML anchor keeps it from turning into copy-paste sprawl:</p>



<pre class="wp-block-code"><code>x-logging: &amp;default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"</code></pre>



<p class="wp-block-paragraph">Be aware that the <code>logging</code> block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with <em>its</em> defaults, which for <code>json-file</code> means straight back to unbounded. Half-specifying is worse than not specifying at all.</p>



<h2 class="wp-block-heading">You deleted the log file and df did not move</h2>



<p class="wp-block-paragraph">Classic 3am mistake, and it is worth understanding rather than memorising.</p>



<p class="wp-block-paragraph">When you <code>rm</code> the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from <code>ls</code>, the daemon keeps writing into a file you can no longer see, and <code>df</code> reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.</p>



<p class="wp-block-paragraph"><code>truncate -s 0</code> is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:</p>



<pre class="wp-block-code"><code>sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"</code></pre>



<p class="wp-block-paragraph">There is a cost. Truncating out from under the daemon can leave an in-flight <code>docker logs -f</code> stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.</p>



<p class="wp-block-paragraph">The same reasoning applies to <code>logrotate</code> rules pointed at <code>/var/lib/docker/containers/*/*.log</code>. Rotating with <code>copytruncate</code> works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker&#8217;s own docs warn against external tools touching those files. If the driver can rotate for you, let it.</p>



<h2 class="wp-block-heading">Choosing a driver, and what each one costs</h2>



<h3 class="wp-block-heading">json-file</h3>



<p class="wp-block-paragraph">The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. <code>docker logs</code> works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.</p>



<h3 class="wp-block-heading">local</h3>



<p class="wp-block-paragraph">Docker&#8217;s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it <em>rotates and compresses by default</em>, retaining five files of 20 MB each per container out of the box. <code>docker logs</code> works exactly as normal. The trade-off is that the format is Docker&#8217;s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is <code>docker logs</code> plus a shipper that talks to the daemon, this is the safest default you can pick.</p>



<h3 class="wp-block-heading">journald</h3>



<p class="wp-block-paragraph">Hands container output to systemd&#8217;s journal, which means you get <code>journalctl</code> filtering, structured fields, and the journal&#8217;s own size management via <code>SystemMaxUse</code> in <code>journald.conf</code>. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.</p>



<p class="wp-block-paragraph">The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at <code>RateLimitIntervalSec</code> and <code>RateLimitBurst</code> before you need them, not after.</p>



<h3 class="wp-block-heading">syslog, fluentd and the remote drivers</h3>



<p class="wp-block-paragraph">These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.</p>



<p class="wp-block-paragraph">What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.</p>



<h3 class="wp-block-heading">none</h3>



<p class="wp-block-paragraph">Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because <code>docker logs</code> returns nothing and you will waste twenty minutes concluding the container is broken.</p>



<h3 class="wp-block-heading">The blocking behaviour worth knowing about</h3>



<p class="wp-block-paragraph">By default the logging path is blocking. If the driver cannot keep up, the container&#8217;s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.</p>



<pre class="wp-block-code"><code>docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1</code></pre>



<p class="wp-block-paragraph">Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.</p>



<h3 class="wp-block-heading">Dual logging, and why docker logs still works</h3>



<p class="wp-block-paragraph">If you switch to a remote driver and find <code>docker logs</code> still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the <code>local</code> driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.</p>



<p class="wp-block-paragraph">Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Reclaiming space when the disk is already full</h2>



<p class="wp-block-paragraph">Order matters here. Free space first, fix the cause second, verify third.</p>



<ol class="wp-block-list">
<li>Confirm logs are actually the problem. Compare <code>du -sh /var/lib/docker/containers</code> against <code>docker system df</code>, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.</li>

<li>Identify the largest log files with the <code>du</code> and <code>sort</code> command from earlier.</li>

<li>Copy anything you need out first, if the logs are part of an active investigation.</li>

<li>Truncate the offenders with <code>truncate -s 0</code>. Never <code>rm</code>.</li>

<li>Set the limits in <code>daemon.json</code>, validate the JSON, restart the daemon.</li>

<li>Recreate the containers so the limits take effect, then confirm with <code>docker inspect</code> on each one.</li>

<li>Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.</li>
</ol>



<p class="wp-block-paragraph">Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Rotation is configured but files still grow past the cap.</strong> The container predates the change. Check <code>docker inspect --format '{{.HostConfig.LogConfig}}'</code> and recreate it.</p>



<p class="wp-block-paragraph"><strong>The daemon will not start after editing daemon.json.</strong> Almost always invalid JSON, or unquoted values in <code>log-opts</code>. Read the actual error with <code>journalctl -u docker --no-pager -n 50</code> rather than guessing.</p>



<p class="wp-block-paragraph"><strong>max-file is set but only one file ever appears.</strong> <code>max-file</code> does nothing without <code>max-size</code>. Set both.</p>



<p class="wp-block-paragraph"><strong>docker logs -f hangs after showing existing output.</strong> Something truncated the file underneath the daemon, usually an external <code>logrotate</code> rule. Reattach to recover, then remove the rule.</p>



<p class="wp-block-paragraph"><strong>docker logs returns nothing at all.</strong> Check the driver. <code>none</code> discards everything, and some remote drivers only serve reads through the dual-logging cache.</p>



<p class="wp-block-paragraph"><strong>Disk usage did not drop after deleting log files.</strong> The daemon still holds the descriptors. Restart the daemon to release the inodes, and use <code>truncate</code> next time.</p>



<p class="wp-block-paragraph"><strong>Logs vanish sooner than the retention you configured.</strong> On <code>journald</code>, check journal rate limiting and <code>SystemMaxUse</code>. On a remote driver in non-blocking mode, check whether the buffer is overflowing.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming a daemon restart applies the new limits to running containers. It does not.</li>

<li>Setting <code>max-file</code> without <code>max-size</code> and believing rotation is on.</li>

<li>Writing numeric values unquoted in <code>log-opts</code>.</li>

<li>Overwriting an existing <code>daemon.json</code> instead of merging, and silently dropping storage or network settings.</li>

<li>Using <code>rm</code> on a live container&#8217;s log file and concluding the disk report is broken.</li>

<li>Specifying a <code>logging.driver</code> in Compose without its <code>options</code>, which resets that service back to the driver&#8217;s own unbounded defaults.</li>

<li>Pointing host <code>logrotate</code> at the Docker containers directory when the driver could have handled it.</li>

<li>Sizing rotation per container without multiplying by container count against the actual disk.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Set logging limits in <code>daemon.json</code> as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.</li>

<li>Prefer the <code>local</code> driver on standalone hosts unless something specifically needs to read the JSON files.</li>

<li>Turn on <code>compress</code>. Log text compresses well enough that you get more retention for the same disk.</li>

<li>Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.</li>

<li>Do the multiplication. Worst-case host usage is <code>max-size</code> × <code>max-file</code> × container count, and it should be a fraction of your disk, not most of it.</li>

<li>Alert on Docker data root usage at a threshold that leaves you time to act.</li>

<li>Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.</li>

<li>Verify with <code>docker inspect</code> after any change. <code>docker info</code> tells you the daemon&#8217;s intent, not what your containers carry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Docker rotate container logs automatically?</h3>



<p class="wp-block-paragraph">Not with the default driver. <code>json-file</code> ships with <code>max-size</code> unlimited and <code>max-file</code> set to 1, so a single file grows without bound. The <code>local</code> driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.</p>



<h3 class="wp-block-heading">Where are Docker container logs stored on the host?</h3>



<p class="wp-block-paragraph">Under the Docker data root, one directory per container, which on a standard Linux install means <code>/var/lib/docker/containers/</code>. Rather than assembling the path by hand, ask the daemon: <code>docker inspect --format '{{.LogPath}}' &lt;container&gt;</code>. That works regardless of a custom <code>data-root</code>.</p>



<h3 class="wp-block-heading">How do I clear Docker logs without stopping the container?</h3>



<p class="wp-block-paragraph">Truncate the file to zero bytes with <code>truncate -s 0</code> on the path from <code>{{.LogPath}}</code>. Do not use <code>rm</code>: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.</p>



<h3 class="wp-block-heading">Why did my daemon.json rotation settings not apply?</h3>



<p class="wp-block-paragraph">Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and <code>docker restart</code> reuses the same container object. Recreate the container, then confirm with <code>docker inspect --format '{{.HostConfig.LogConfig}}'</code>.</p>



<h3 class="wp-block-heading">Should I use the local driver or json-file?</h3>



<p class="wp-block-paragraph">Use <code>local</code> if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use <code>json-file</code> if an agent or shipper on the host tails those files, because it cannot parse the <code>local</code> format. Either way <code>docker logs</code> behaves the same.</p>



<h3 class="wp-block-heading">Can I use logrotate for Docker container logs instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of hosts do, but the driver options are the better tool. A <code>copytruncate</code> rule pointed at the containers directory reaches into files the daemon treats as private, and can leave <code>docker logs -f</code> streams stalled after each rotation. Use <code>logrotate</code> for logs your applications write to mounted volumes, and let the driver handle stdout.</p>



<h3 class="wp-block-heading">How much disk should I budget for container logs?</h3>



<p class="wp-block-paragraph">Worst case is <code>max-size</code> × <code>max-file</code> per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.</p>



<p class="wp-block-paragraph">So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with <code>docker inspect</code> rather than trusting <code>docker info</code>. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with container logging on your hosts?</h2>



<p class="wp-block-paragraph">Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits</li>

<li>Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number</li>

<li>Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back</li>

<li>Choosing between <code>json-file</code>, <code>local</code>, <code>journald</code> and shipping off-box, including the blocking behaviour trade-offs under load</li>

<li>Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends</li>

<li>Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me your <code>daemon.json</code>, the output of <code>docker ps</code> and a <code>du -sh</code> of your containers directory, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-log-rotation/">Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
