<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>SSH | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/ssh/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/ssh/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 31 Oct 2026 06:08:13 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>SSH | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/ssh/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</title>
		<link>https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 31 Oct 2026 06:08:13 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Chain of Custody]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Digital Forensics]]></category>
		<category><![CDATA[File Integrity Monitoring]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Persistence Mechanisms]]></category>
		<category><![CDATA[Rescue Mode]]></category>
		<category><![CDATA[Rootkits]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[systemd Timers]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Shell]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=496</guid>

					<description><![CDATA[<p>Killing the process removes the symptom. It does nothing about the SSH key on a service account or the timer pulling a fresh payload every twenty minutes. A working guide to compromised VPS recovery: how to judge the depth of an intrusion, keep the evidence, enumerate the access paths and persistence people miss, and decide honestly whether you can clean in place or need to rebuild.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email from the provider is short and unfriendly. Outbound scanning detected on port 22, network access to your instance has been rate limited, respond within 24 hours. You SSH in, find a process you don&#8217;t recognise chewing a core, kill it, delete the binary out of <code>/tmp</code>, and write back saying it&#8217;s handled.</p>



<p class="wp-block-paragraph">Three days later you get the same email.</p>



<p class="wp-block-paragraph">That gap is the whole problem with compromised VPS recovery. Killing the process removes the symptom. It does nothing about the SSH key that got added to a service account, the systemd timer that pulls a fresh payload every twenty minutes, or the reason they got in the first time. You cleaned up artifacts. You never closed the access.</p>



<p class="wp-block-paragraph">This post is about what you can honestly fix in place, what you can&#8217;t, and how to tell the difference before you sink a weekend into a box you&#8217;re going to rebuild anyway: evidence handling, the access paths and persistence people miss, why package verification proves less than it looks, and the one in-place technique that genuinely holds up.</p>



<h2 class="wp-block-heading">What compromised VPS recovery actually means</h2>



<p class="wp-block-paragraph">There are two goals hiding inside &#8220;recover the server&#8221; and they pull in different directions.</p>



<ul class="wp-block-list"><li><strong>Restore service.</strong> Get the site back, stop the abuse traffic, keep the client happy. Fast, and you can do it on the same instance.</li><li><strong>Restore trust.</strong> Be able to say, with something behind it, that nothing on this machine is executing code you didn&#8217;t put there. Slow, and on a machine where someone had root, often impossible.</li></ul>



<p class="wp-block-paragraph">A reinstall is not a technical superstition. It&#8217;s the cheapest way to buy the second goal. You throw away every byte you can&#8217;t account for and start from a base image plus config you control. Cleaning in place buys you the first goal quickly and the second one slowly, sometimes never.</p>



<p class="wp-block-paragraph">So the default answer is still: rebuild from a known-good image, restore validated data, re-point DNS. What follows is for when that isn&#8217;t available today, or when you have to investigate before you&#8217;re allowed to destroy anything.</p>



<h2 class="wp-block-heading">Decide this first: how deep did they get</h2>



<p class="wp-block-paragraph">Everything else follows from this. The depth of the compromise decides whether in-place recovery is defensible or wishful thinking.</p>



<h3 class="wp-block-heading">Unprivileged application code</h3>



<p class="wp-block-paragraph">A web shell dropped in a document root, running as the PHP-FPM pool user or <code>www-data</code>. It can read what that user reads, write where that user writes, and make outbound connections. It cannot touch <code>/usr/bin</code> or install a kernel module.</p>



<p class="wp-block-paragraph">In-place recovery is reasonable here, as long as you can prove the blast radius stayed inside that user. That proof is the hard part, not the cleanup.</p>



<h3 class="wp-block-heading">A service account with a path upward</h3>



<p class="wp-block-paragraph">Same as above, but the account has a sudoers entry, write access to a systemd unit directory, membership in the <code>docker</code> group, or a writable script that root runs on a schedule. Treat this as a root compromise until you&#8217;ve disproved it, because all four of those are one command away from root.</p>



<h3 class="wp-block-heading">Root</h3>



<p class="wp-block-paragraph">Now the machine can lie to you. Root can replace <code>ps</code>, edit the package database, truncate logs, and set the immutable attribute on files so your own cleanup silently fails. Anything the live system tells you is a claim, not a fact.</p>



<p class="wp-block-paragraph">You can still investigate. You cannot honestly certify the result. If someone needs a clean bill of health, rebuild.</p>



<h3 class="wp-block-heading">Kernel</h3>



<p class="wp-block-paragraph">A loadable kernel module or an eBPF program that hides processes, files and network connections from userspace. Your tools return whatever the rootkit decides they should return. There is no in-place recovery here, only offline analysis of the disk and a rebuild.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Before you touch anything, keep the evidence</h2>



<p class="wp-block-paragraph">The first fifteen minutes decide how much you&#8217;ll be able to learn later. Most of what tells you how bad it is only exists while the box is running, and most of it gets destroyed by the instinctive reaction.</p>



<ol class="wp-block-list"><li><strong>Isolate at the edge, not from inside.</strong> Use your provider&#8217;s firewall or security group to cut everything except your own source address. Running <code>ip link set eth0 down</code> from an SSH session locks you out and does not stop a local timer from running.</li><li><strong>Take a snapshot.</strong> Every serious VPS provider offers one, and it costs almost nothing to keep for a couple of weeks. This is your only chance at a copy of the disk in its compromised state. Providers like Hetzner, Contabo, OVHcloud and InterServer all expose snapshots or images from the control panel.</li><li><strong>Collect volatile state before you reboot.</strong> Sockets, processes, open files and the mapping from process to on-disk binary all disappear the moment the box restarts.</li><li><strong>Copy the collection off the box.</strong> To your workstation or a bucket, not to another directory on the same instance.</li></ol>



<p class="wp-block-paragraph">A collection pass that fits in a couple of minutes:</p>



<pre class="wp-block-code"><code># Sockets with owning process, numeric, TCP and UDP
ss -tunap &gt; /tmp/ir-sockets.txt

# Process tree, full command lines
ps auxf &gt; /tmp/ir-ps.txt

# Where each PID's executable actually lives on disk.
# A deleted binary still running shows up as "(deleted)".
for p in /proc/[0-9]*; do
  echo "$p -&gt; $(readlink "$p/exe" 2&gt;/dev/null)"
done &gt; /tmp/ir-exe.txt

# Logins, including failed ones
last -F -f /var/log/wtmp &gt; /tmp/ir-last.txt
lastb -F                 &gt; /tmp/ir-lastb.txt 2&gt;/dev/null

# Journal for the suspect window
journalctl --since "7 days ago" --no-pager &gt; /tmp/ir-journal.txt</code></pre>



<p class="wp-block-paragraph">The <code>readlink</code> loop is the one people skip and then wish they hadn&#8217;t. A process whose executable was unlinked after launch is a strong signal on its own, and it&#8217;s the only place the original path survives.</p>



<p class="wp-block-paragraph">If <code>auditd</code> was already running, pull its logs too. If it wasn&#8217;t, note that for the rebuild rather than installing it now on a machine you don&#8217;t trust.</p>



<h2 class="wp-block-heading">Failure family 1: you removed the payload and left the door</h2>



<p class="wp-block-paragraph">This is the one that produces the second abuse email. Access mechanisms are boring, they don&#8217;t show up in <code>top</code>, and they survive every cleanup that focuses on running processes.</p>



<pre class="wp-block-code"><code># Authorized keys for every account that has a home directory
for h in $(getent passwd | cut -d: -f6 | sort -u); do
  [ -f "$h/.ssh/authorized_keys" ] &amp;&amp; echo "== $h" &amp;&amp; cat "$h/.ssh/authorized_keys"
done

# sshd may have been told to read keys from somewhere else entirely
grep -Ei 'AuthorizedKeysFile|AuthorizedKeysCommand|PermitRootLogin|PasswordAuthentication|^Match' 
  /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2&gt;/dev/null

# Any account with UID 0 other than root
awk -F: '($3 == 0) {print}' /etc/passwd

# Accounts that can actually log in
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd

# Sudo grants, including drop-in files
cat /etc/sudoers; ls -la /etc/sudoers.d/</code></pre>



<p class="wp-block-paragraph">Two of these deserve more attention than they usually get. <code>AuthorizedKeysFile</code> pointing at a path outside a user&#8217;s home means your audit of <code>~/.ssh/authorized_keys</code> proves nothing. And a <code>Match</code> block near the bottom of <code>sshd_config</code> can quietly re-enable password authentication or root login for one user or one source address while the top of the file still looks locked down.</p>



<p class="wp-block-paragraph">Also check <code>/etc/pam.d/</code> for modules you don&#8217;t recognise. PAM is a legitimate extension point, which is exactly why it&#8217;s a good hiding place, and a single added line can accept any password for a given account.</p>



<h2 class="wp-block-heading">Failure family 2: persistence in the places you didn&#8217;t check</h2>



<p class="wp-block-paragraph">Cron gets checked. The rest usually doesn&#8217;t. MITRE ATT&amp;CK catalogues these as separate techniques for a reason: they&#8217;re independent, and finding one tells you nothing about the others.</p>



<pre class="wp-block-code"><code># Per-user crontabs and the system drop-in directories
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2&gt;/dev/null; done
ls -la /etc/cron.d/ /etc/cron.hourly/ /etc/cron.daily/ /var/spool/cron/

# systemd: enabled units and every timer, including inactive ones
systemctl list-unit-files --type=service --state=enabled
systemctl list-timers --all
systemctl cat suspicious-unit.service   # shows the actual ExecStart

# Per-user systemd units survive logout when lingering is on
ls -la /root/.config/systemd/user/ /home/*/.config/systemd/user/ 2&gt;/dev/null
loginctl list-users

# One-shot at jobs, frequently overlooked
atq

# Dynamic linker hijacking
cat /etc/ld.so.preload 2&gt;/dev/null

# Shell and login hooks
ls -la /etc/profile.d/ /etc/update-motd.d/ 2&gt;/dev/null
grep -rn 'curl|wget|base64|/dev/tcp' /root/.bashrc /home/*/.bashrc 
  /etc/bash.bashrc 2&gt;/dev/null

# Unexpected setuid binaries and file capabilities
find / -xdev -perm -4000 -type f 2&gt;/dev/null
getcap -r / 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph"><code>/etc/ld.so.preload</code> is worth understanding rather than just grepping. Any library listed there is loaded into every dynamically linked process before anything else, which lets it intercept ordinary libc calls. That&#8217;s how a userspace rootkit makes <code>readdir</code> skip its own files and <code>ps</code> skip its own processes without touching the kernel. If that file exists on a machine where you didn&#8217;t put it, everything you&#8217;ve observed so far is suspect.</p>



<p class="wp-block-paragraph">The other one to internalise is timers. A cleanup that looks successful, followed by reinfection at a suspiciously round interval, is almost always a systemd timer or a cron entry you didn&#8217;t enumerate. <code>systemctl list-timers --all</code> matters because a timer that isn&#8217;t currently active is invisible without the flag.</p>



<h2 class="wp-block-heading">Failure family 3: verification that verifies less than you think</h2>



<p class="wp-block-paragraph">Package managers keep hashes of the files they installed, so you can ask the distro whether anything on disk drifted from what shipped.</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky, Fedora: verify every installed package
rpm -Va

# Debian and Ubuntu: dpkg's own verify, no extra package needed
dpkg -V

# Optional on Debian/Ubuntu, checks more files than dpkg -V
debsums -s</code></pre>



<p class="wp-block-paragraph">In <code>rpm -Va</code> output, the character positions are the checks that failed: <code>5</code> is a changed digest, <code>S</code> size, <code>T</code> timestamp, <code>M</code> mode. A <code>c</code> after the flags marks a config file, and config files change legitimately all the time. A modified digest on something in <code>/usr/bin</code> or <code>/usr/sbin</code> does not.</p>



<p class="wp-block-paragraph">Now the caveats, because this is where people talk themselves into a false clean bill of health:</p>



<ul class="wp-block-list"><li>It only covers files that came from a package. A web shell in your document root, a binary in <code>/tmp</code>, a rogue systemd unit and anything you compiled yourself are all invisible to it.</li><li>The database and the verifying binary both live on the machine being verified. Root can edit both.</li><li>Timestamps and sizes are trivially restorable. A digest change with an unchanged size and timestamp is a specific tell that someone tried to cover their tracks.</li></ul>



<p class="wp-block-paragraph">Treat findings as strong evidence and a clean result as weak evidence. That asymmetry is the useful part.</p>



<h3 class="wp-block-heading">Application-level checksums</h3>



<p class="wp-block-paragraph">For anything running WordPress, WP-CLI covers the gap the package manager leaves, because it verifies against checksums published by wordpress.org rather than anything stored locally:</p>



<pre class="wp-block-code"><code>wp core verify-checksums
wp plugin verify-checksums --all</code></pre>



<p class="wp-block-paragraph">Neither command covers themes or premium plugins that aren&#8217;t in the public repository, so pair them with a modification-time sweep of the document root.</p>



<h3 class="wp-block-heading">Rootkit scanners</h3>



<p class="wp-block-paragraph">The honest case for <code>rkhunter</code> and <code>chkrootkit</code>: they&#8217;re free, they run in a minute, and they catch known families and common mistakes that an experienced pair of eyes might skim past. Running one costs you nothing.</p>



<p class="wp-block-paragraph">The limit: they&#8217;re signature and heuristic based, they check for kits that were known when the signature set was written, and they ask the same possibly compromised kernel and libc that everything else asks. A clean scan means the obvious things aren&#8217;t there. It is not a negative result.</p>



<h2 class="wp-block-heading">Failure family 4: the way in is still open</h2>



<p class="wp-block-paragraph">If you finish the cleanup without an answer to &#8220;how did they get in&#8221;, you haven&#8217;t finished. You&#8217;ve just reset the clock.</p>



<p class="wp-block-paragraph">The practical technique is to find the earliest artifact you&#8217;re confident about, then read the access logs around that timestamp. Attackers rarely land quietly on the first attempt, and the noise before the successful request is usually more legible than the request itself.</p>



<pre class="wp-block-code"><code># Files in the document root modified in the last two weeks
find /var/www -xdev -type f -newermt '14 days ago' -printf '%T+ %pn' 2&gt;/dev/null | sort

# Same idea for the usual drop locations
find /tmp /var/tmp /dev/shm -xdev -type f -newermt '14 days ago' -ls 2&gt;/dev/null</code></pre>



<p class="wp-block-paragraph">The usual suspects, roughly in the order I check them: an outdated plugin or CMS component with a public exploit, an admin panel exposed to the whole internet, a weak or reused password on an account with a shell, a deploy key or CI token leaked into a repository, and a service bound to <code>0.0.0.0</code> that was only ever meant to be local. Redis and an unauthenticated Docker socket are the two that come up most in that last category.</p>



<p class="wp-block-paragraph">If your logs for the relevant window are gone, that absence is itself a finding. Shipping logs off-box to somewhere append-only, whether that&#8217;s a hosted platform or your own Loki instance, is the single change that would have made this investigation tractable. Worth noting for the rebuild.</p>



<h2 class="wp-block-heading">The in-place technique that actually holds up</h2>



<p class="wp-block-paragraph">Everything above shares a weakness: you&#8217;re asking a possibly compromised system to describe itself. There&#8217;s one way around that which doesn&#8217;t require a reinstall, and it&#8217;s the reason I don&#8217;t dismiss in-place work entirely.</p>



<p class="wp-block-paragraph">Boot the VPS into your provider&#8217;s rescue image, or attach a snapshot of the disk to a second clean instance. Then mount the target filesystem read-only and inspect it from a kernel the attacker never touched.</p>



<pre class="wp-block-code"><code># In the rescue environment: identify the target disk
lsblk -f

# Mount read-only. On ext4 add noload, on XFS add norecovery,
# so the kernel doesn't replay the journal and write to the disk.
mkdir -p /mnt/target
mount -o ro,noload /dev/vdb1 /mnt/target      # ext4
# mount -o ro,norecovery /dev/vdb1 /mnt/target  # xfs

# Inspect without chrooting into it
ls -la /mnt/target/etc/cron.d/
cat /mnt/target/etc/ld.so.preload 2&gt;/dev/null
find /mnt/target/etc/systemd/system -newermt '30 days ago' -ls</code></pre>



<p class="wp-block-paragraph">Why this is different in kind, not just degree: a userspace rootkit hooking libc, or a kernel module hiding directory entries, works by intercepting calls on the running system. Mounted from outside, those files are just bytes. Nothing gets to filter what you see.</p>



<p class="wp-block-paragraph">Two things to keep straight. Don&#8217;t <code>chroot</code> into the mounted filesystem and start running its binaries, because you&#8217;ve just handed control back to the attacker&#8217;s code. And read-only isn&#8217;t automatic: mounting a dirty ext4 or XFS filesystem will replay the journal and write to the device unless you pass the option that suppresses it, which is enough to invalidate the copy if anyone later cares about chain of custody.</p>



<h2 class="wp-block-heading">When your checks disagree</h2>



<ul class="wp-block-list"><li><strong>Package verification is clean but the behaviour continues.</strong> The payload isn&#8217;t in a packaged file. Look at the document root, <code>/opt</code>, <code>/usr/local</code>, and anything in a home directory. That&#8217;s most of the filesystem, and none of it is covered.</li><li><strong>A process has no matching binary on disk.</strong> The executable was unlinked after launch. Check <code>ls -l /proc/PID/exe</code> for the <code>(deleted)</code> suffix and the original path, then find what restarts it, because something must.</li><li><strong>A file refuses to be deleted as root.</strong> Check <code>lsattr</code> for the immutable attribute and clear it with <code>chattr -i</code>. Attackers use this specifically because it makes casual cleanup fail silently.</li><li><strong>Rebooting appears to fix it and it comes back on a schedule.</strong> Enumerate every timer and cron path again, including <code>systemctl list-timers --all</code> and per-user units under lingering sessions.</li><li><strong>Journal entries are missing for the interesting window.</strong> Run <code>journalctl --verify</code>. Truncation is a finding, not a dead end, and it argues for moving to the offline path.</li><li><strong>The outbound traffic continues after you&#8217;ve killed everything.</strong> You&#8217;re either looking at a hidden process or a second compromised container on the same host. Confirm from the provider&#8217;s traffic graphs rather than from tools on the box.</li></ul>



<h2 class="wp-block-heading">Arguments for staying in place that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;The rootkit scanner came back clean.&#8221;</strong> It checked for known kits using tools running on the suspect system. Absence of evidence, and weak absence at that.</li><li><strong>&#8220;I rotated all the passwords.&#8221;</strong> Passwords are one access path. SSH keys, API tokens, sudo rules, PAM modules and scheduled jobs are the others, and none of them care about your password reset.</li><li><strong>&#8220;It&#8217;s only a web shell, the OS is fine.&#8221;</strong> Sometimes true, and it&#8217;s the one case where in-place recovery is genuinely defensible. But it&#8217;s a conclusion you have to earn by ruling out privilege escalation, not an assumption you start from.</li><li><strong>&#8220;I restored from backup.&#8221;</strong> Restoring content doesn&#8217;t remove a key from <code>/root/.ssh</code>, and if the backup predates your detection by less than the dwell time, you&#8217;ve restored the compromise. Know when the earliest artifact appeared before you pick a restore point.</li><li><strong>&#8220;I don&#8217;t have time to rebuild.&#8221;</strong> The honest version of this is a real constraint, not a technical argument. But if your infrastructure is described in code and your data restore is tested, a rebuild is often faster than the second cleanup, never mind the third.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list"><li><strong>Snapshot first, always.</strong> Before any judgement call. It&#8217;s cheap and it&#8217;s the only irreversible decision in the whole process.</li><li><strong>Isolate at the provider edge</strong> and collect volatile state while the box is still up.</li><li><strong>Establish depth.</strong> Did anything run as root? If you can&#8217;t answer that with evidence, assume yes.</li><li><strong>If root is on the table, plan the rebuild now</strong> and treat the in-place work as investigation rather than remediation. The two goals stop competing once you&#8217;ve decided which one you&#8217;re pursuing.</li><li><strong>If it stayed inside one unprivileged account,</strong> clean in place: remove the payload, close the access paths, patch the entry point, rotate every credential that account could read.</li><li><strong>Do the offline read-only pass anyway.</strong> It&#8217;s twenty minutes in a rescue image and it&#8217;s the only evidence that doesn&#8217;t depend on the compromised machine&#8217;s cooperation.</li><li><strong>Rebuild the hardening on the new instance,</strong> not the old one. Key-only SSH, an allowlist on the admin port, automatic security updates, off-box logging, and file integrity monitoring with AIDE or a platform like Wazuh, so the next incident starts with a baseline instead of guesswork.</li></ol>



<p class="wp-block-paragraph">On that allowlist point: restricting SSH to a fixed source address is the single highest-value control here, and it&#8217;s the one people skip because their own IP changes. A VPN with a dedicated IP, which providers like NordVPN and Surfshark both offer, or a Cloudflare Tunnel in front of the admin path, gives you a stable origin to allow without exposing the port to the internet.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you ever recover a hacked Linux server without reinstalling?</h3>



<p class="wp-block-paragraph">Yes, when the compromise stayed inside an unprivileged application account and you can demonstrate that it did. Once anything ran as root, you can restore service in place but you cannot honestly certify the machine is clean, because every tool you&#8217;d use to check runs under the code you&#8217;re checking for.</p>



<h3 class="wp-block-heading">Should I reboot the VPS as soon as I notice something wrong?</h3>



<p class="wp-block-paragraph">Not first. A reboot destroys the process list, open sockets, and the mapping from running processes to deleted binaries, which is often the most informative evidence available. Snapshot, isolate at the edge, collect volatile state, then reboot if you need to.</p>



<h3 class="wp-block-heading">How long do attackers usually stay before I notice?</h3>



<p class="wp-block-paragraph">Long enough that you should be suspicious of any restore point close to your detection date. Rather than trusting a published average, find the earliest artifact you&#8217;re confident about on your own machine, then add margin. If your logs don&#8217;t reach back that far, that&#8217;s a finding in itself.</p>



<h3 class="wp-block-heading">Is restoring from a backup enough after a compromise?</h3>



<p class="wp-block-paragraph">Only if the backup predates the intrusion and covers everything that was modified. A content-only restore leaves SSH keys, cron entries, systemd units and package changes in place, so it usually removes the visible damage while preserving the access. Restore data onto a fresh instance instead of over a compromised one.</p>



<h3 class="wp-block-heading">What&#8217;s the fastest way to check for a compromised VPS?</h3>



<p class="wp-block-paragraph">Sockets with owning processes (<code>ss -tunap</code>), authorized keys across every home directory, every timer including inactive ones (<code>systemctl list-timers --all</code>), <code>/etc/ld.so.preload</code>, and package verification. That&#8217;s a few minutes and it catches the majority of commodity intrusions. It will not catch a competent one.</p>



<h3 class="wp-block-heading">Do I need to tell my hosting provider?</h3>



<p class="wp-block-paragraph">If they contacted you about abuse traffic, yes, and quickly, because unanswered abuse reports lead to suspension. It&#8217;s also worth asking whether they can supply traffic graphs for the affected period. That data comes from outside your instance, which makes it more trustworthy than anything on it.</p>



<h3 class="wp-block-heading">If there was personal data on the server, does that change anything?</h3>



<p class="wp-block-paragraph">It changes the order of operations. Preserve the snapshot before you clean anything, document what you did and when, and check your notification obligations before the technical work absorbs your attention. Cleanup that destroys evidence is a much bigger problem when the incident is reportable.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Compromised VPS recovery isn&#8217;t really a cleanup procedure. It&#8217;s a decision about what you&#8217;re able to claim afterwards.</p>



<p class="wp-block-paragraph">If the intrusion stayed inside an unprivileged account and you can show it, clean in place and move on. If root was ever involved, every check you run on that box is a statement from a machine with a motive. Take the snapshot, do the offline read-only pass because it&#8217;s the only view the attacker doesn&#8217;t control, and put the rebuilt hardening on a fresh instance rather than the one you&#8217;ve been arguing with.</p>



<p class="wp-block-paragraph">And find the way in. A cleanup that doesn&#8217;t end with a patched entry point isn&#8217;t a recovery, it&#8217;s an interval.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on a compromised server?</h2>



<p class="wp-block-paragraph">Incident work is unpleasant to do alone and easy to do in the wrong order. This is the kind of thing I help with:</p>



<ul class="wp-block-list"><li>Triage on a live VPS: establishing depth of compromise, collecting volatile evidence before it&#8217;s lost, and giving you a straight answer on clean-in-place versus rebuild</li><li>Offline analysis of a snapshot or rescue-mode disk, including timeline reconstruction from filesystem timestamps and whatever logs survived</li><li>Full enumeration of access paths and persistence: SSH configuration, PAM, sudoers, cron, systemd units and timers, preload hooks and setuid binaries</li><li>Finding the entry point rather than just the payload, correlating web and auth logs against the earliest artifacts on disk</li><li>Rebuilding the server properly afterwards, in code, with key-only SSH, an admin allowlist, off-box logging and file integrity monitoring in place from day one</li><li>WordPress and PHP application cleanup where the compromise came in through the stack rather than the OS</li></ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one right now, send me the output of <code>ss -tunap</code>, your <code>systemctl list-timers --all</code>, and the abuse report if you got one. That&#8217;s usually enough to tell you how deep this goes.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/compromised-vps-recovery-without-reinstall/">Compromised VPS Recovery: What You Can Actually Fix Without Reinstalling</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</title>
		<link>https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Ansible Collections]]></category>
		<category><![CDATA[Ansible Vault]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=204</guid>

					<description><![CDATA[<p>A single Ansible run can take a blank VPS to a hardened, reproducible server. The hard part is not the tasks, it is the order. Here is the ordering that keeps you logged in, the failures that pass green and bite two weeks later, and how to make the second run as safe as the first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.</p>



<p class="wp-block-paragraph">That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report <code>changed</code>, do nothing useful, and leave you believing the server is hardened when it is not.</p>



<p class="wp-block-paragraph">This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.</p>



<h2 class="wp-block-heading">What a one-run bootstrap actually has to survive</h2>



<p class="wp-block-paragraph">A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.</p>



<p class="wp-block-paragraph">Four things break it, and they break it in different ways:</p>



<ul class="wp-block-list">
<li><strong>The chicken-and-egg problem.</strong> Ansible needs Python on the target to run almost any module, and a minimal image may not have it.</li>

<li><strong>Lockout.</strong> You remove your own access before proving the replacement access works.</li>

<li><strong>Non-idempotent tasks.</strong> The first run is fine. The second run is where the damage happens.</li>

<li><strong>Missing collections.</strong> The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with <code>ansible-core</code>.</li>
</ul>



<p class="wp-block-paragraph">Everything below is organised around those four, because that is how they actually show up.</p>



<h2 class="wp-block-heading">The chicken-and-egg problem: no Python, no modules</h2>



<p class="wp-block-paragraph">Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.</p>



<p class="wp-block-paragraph">The escape hatch is <code>ansible.builtin.raw</code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with <code>gather_facts: false</code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.</p>



<pre class="wp-block-code"><code>- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update &amp;&amp; apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:</code></pre>



<p class="wp-block-paragraph">Two details worth understanding rather than copying. <code>failed_when: false</code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And <code>ansible.builtin.setup</code> is the fact-gathering module called explicitly, which means from that point on <code>ansible_distribution</code> and friends are available even though the play started with gathering disabled.</p>



<p class="wp-block-paragraph">If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.</p>



<h2 class="wp-block-heading">Failure family one: locking yourself out while the playbook says ok</h2>



<p class="wp-block-paragraph">This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.</p>



<h3 class="wp-block-heading">The order that survives</h3>



<p class="wp-block-paragraph">The rule is simple: <strong>nothing that removes access happens until something has proven the new access works.</strong> In practice:</p>



<ol class="wp-block-list">
<li>Connect as whatever the provider handed you, usually <code>root</code> or a sudo-capable default user.</li>

<li>Create the admin account, add it to the sudo or wheel group, install its authorized key.</li>

<li>Write the sudoers drop-in and validate it with <code>visudo -cf</code>.</li>

<li><strong>Prove you can log in as that account and escalate.</strong> This is the step everyone skips.</li>

<li>Install the firewall and allow SSH <em>before</em> enabling it.</li>

<li>Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.</li>

<li>Only now disable root login and password authentication.</li>
</ol>



<p class="wp-block-paragraph">Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:</p>



<pre class="wp-block-code"><code>- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"</code></pre>



<p class="wp-block-paragraph">Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.</p>



<h3 class="wp-block-heading">The socket activation trap</h3>



<p class="wp-block-paragraph">This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. <code>ssh.socket</code> owns the listening port, and <code>Port</code> in <code>sshd_config</code> is not what decides where the daemon listens.</p>



<p class="wp-block-paragraph">So your playbook writes <code>Port 2222</code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.</p>



<p class="wp-block-paragraph">The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:</p>



<pre class="wp-block-code"><code># Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'</code></pre>



<p class="wp-block-paragraph">If <code>ssh.socket</code> is enabled, the override needs an empty <code>ListenStream=</code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.</p>



<p class="wp-block-paragraph">Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.</p>



<h3 class="wp-block-heading">Firewall ordering</h3>



<p class="wp-block-paragraph">UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an <code>UNREACHABLE</code> that will not recover. Rules go in first, always:</p>



<pre class="wp-block-code"><code>- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled</code></pre>



<p class="wp-block-paragraph">Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.</p>



<h2 class="wp-block-heading">Failure family two: the second run is the dangerous one</h2>



<p class="wp-block-paragraph">A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are <code>shell</code> and <code>command</code> tasks with no guard.</p>



<p class="wp-block-paragraph">An unguarded <code>shell</code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.</p>



<p class="wp-block-paragraph">Guard them properly:</p>



<ul class="wp-block-list">
<li><code>creates:</code> or <code>removes:</code> makes the task skip when the target state already exists. Prefer this.</li>

<li><code>changed_when: false</code> for read-only commands so they stop lying about changing things.</li>

<li><code>when:</code> driven by a registered check, for anything conditional.</li>
</ul>



<p class="wp-block-paragraph">The other idempotency trap is <code>lineinfile</code> against <code>sshd_config</code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:</p>



<pre class="wp-block-code"><code>- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false</code></pre>



<p class="wp-block-paragraph">Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so <code>validate:</code> on the template task with <code>sshd -t -f %s</code> will fail on a temporary file that has no host keys and no context. Run <code>sshd -t</code> as its own task afterwards instead, and let the handler restart only if it passes.</p>



<p class="wp-block-paragraph">Two prerequisites for the drop-in approach. The base <code>sshd_config</code> needs an <code>Include /etc/ssh/sshd_config.d/*.conf</code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the <em>first</em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.</p>



<h2 class="wp-block-heading">Failure family three: the collections are not where you think</h2>



<p class="wp-block-paragraph"><code>ansible-core</code> ships the <code>ansible.builtin</code> namespace and essentially nothing else. The full <code>ansible</code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.</p>



<p class="wp-block-paragraph">Things people assume are built in and are not: <code>ufw</code> and <code>timezone</code> live in <code>community.general</code>. <code>authorized_key</code>, <code>sysctl</code> and <code>mount</code> live in <code>ansible.posix</code>. Declare them and pin them:</p>



<pre class="wp-block-code"><code># requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml</code></pre>



<p class="wp-block-paragraph">Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. <code>ansible-lint</code> flags this and it is worth listening to.</p>



<h2 class="wp-block-heading">Failure family four: secrets in the repository</h2>



<p class="wp-block-paragraph">Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.</p>



<p class="wp-block-paragraph">Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:</p>



<pre class="wp-block-code"><code>ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass</code></pre>



<p class="wp-block-paragraph">The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Putting it together: how to bootstrap a VPS with Ansible end to end</h2>



<p class="wp-block-paragraph">Structure matters less than order, but a shape that works well is a single <code>site.yml</code> calling roles in dependency order, with the risky access changes fenced off at the end.</p>



<pre class="wp-block-code"><code># inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22</code></pre>



<p class="wp-block-paragraph">Then the run itself, in this order:</p>



<ol class="wp-block-list">
<li><strong>Reachability.</strong> Python check, fact gathering, timezone, hostname, package cache refresh.</li>

<li><strong>Identity.</strong> Admin user, group membership, authorized key, sudoers drop-in with <code>visudo -cf</code> validation.</li>

<li><strong>Verification gate.</strong> The task shown earlier. Nothing destructive runs before this passes.</li>

<li><strong>Firewall.</strong> Rules first, policy second, enable third.</li>

<li><strong>Unattended updates.</strong> <code>unattended-upgrades</code> on Debian and Ubuntu, <code>dnf-automatic</code> with its systemd timer on RHEL-family hosts.</li>

<li><strong>Brute-force protection.</strong> fail2ban with your own <code>/etc/fail2ban/jail.local</code>, never edits to <code>jail.conf</code>.</li>

<li><strong>Workload.</strong> Docker, a web server, a monitoring agent, whatever the box is for.</li>

<li><strong>Access teardown.</strong> Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.</li>
</ol>



<p class="wp-block-paragraph">When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:</p>



<pre class="wp-block-code"><code>- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60</code></pre>



<p class="wp-block-paragraph"><code>meta: reset_connection</code> does not accept a <code>when</code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.</p>



<h2 class="wp-block-heading">Test it before you point it at anything real</h2>



<p class="wp-block-paragraph">The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.</p>



<pre class="wp-block-code"><code># catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff</code></pre>



<p class="wp-block-paragraph">One caveat on <code>--check</code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong><code>UNREACHABLE</code> on the very first task.</strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with <code>ssh-keygen -R</code> and re-add the correct one with <code>ssh-keyscan</code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.</li>

<li><strong><code>UNREACHABLE</code> immediately after a firewall task.</strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.</li>

<li><strong>&#8220;Missing sudo password&#8221;.</strong> The sudoers drop-in is not in effect or has bad syntax. Validate with <code>visudo -cf</code> on the file itself before trusting it.</li>

<li><strong>Module not found under an FQCN.</strong> The collection is not installed on the machine running the playbook. Confirm with <code>ansible-galaxy collection list</code>.</li>

<li><strong>New group membership ignored.</strong> Adding a user to <code>docker</code> mid-play does not affect the already-open connection. Insert <code>meta: reset_connection</code> after the group change.</li>

<li><strong>fail2ban installed but not banning.</strong> On images without rsyslog, <code>/var/log/auth.log</code> may not exist, so a jail configured to read it never starts. Check <code>fail2ban-client status</code> and switch the jail to the systemd backend if the log file is absent.</li>

<li><strong>Everything green, port unchanged.</strong> Socket activation. Check <code>systemctl is-enabled ssh.socket</code> and <code>ss -tlnp</code>.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling root login in the same play that creates the replacement user, with no verification step between them.</li>

<li>Enabling UFW before the SSH allow rule exists.</li>

<li>Assuming <code>Port</code> in <code>sshd_config</code> is authoritative on every distribution.</li>

<li>Unguarded <code>shell</code> tasks that notify handlers, causing service restarts on every run.</li>

<li>Editing <code>jail.conf</code> instead of creating <code>jail.local</code>, so the next package update silently reverts it.</li>

<li>Committing plaintext secrets because &#8220;it is only a bootstrap playbook&#8221;.</li>

<li>Testing on a host you cannot afford to lose, with no console access configured.</li>

<li>Relying on <code>--check</code> as a substitute for a real run on a disposable host.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.</li>

<li>Put a verification task before every irreversible one, and let the play fail there rather than after.</li>

<li>Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.</li>

<li>Validate configuration before restarting the service that reads it. <code>sshd -t</code>, <code>visudo -cf</code>, <code>nginx -t</code>.</li>

<li>Pin collections in <code>requirements.yml</code> and install them in CI, not just locally.</li>

<li>Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.</li>

<li>Reboot once during testing. Half of the silent failures only appear after a restart.</li>

<li>Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you really bootstrap a VPS with Ansible in a single playbook run?</h3>



<p class="wp-block-paragraph">Yes, including changing the SSH user and port mid-run. The mechanism is <code>set_fact</code> to update the connection variables, <code>meta: reset_connection</code> to drop the cached connection, and <code>wait_for_connection</code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.</p>



<h3 class="wp-block-heading">Should I use cloud-init or Ansible for initial server setup?</h3>



<p class="wp-block-paragraph">Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.</p>



<h3 class="wp-block-heading">Do I need Terraform or OpenTofu as well?</h3>



<p class="wp-block-paragraph">Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.</p>



<h3 class="wp-block-heading">Why does my playbook work the first time and fail the second?</h3>



<p class="wp-block-paragraph">Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.</p>



<h3 class="wp-block-heading">Is changing the SSH port worth doing?</h3>



<p class="wp-block-paragraph">It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.</p>



<h3 class="wp-block-heading">How do I recover if the playbook locks me out?</h3>



<p class="wp-block-paragraph">Through the provider&#8217;s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.</p>



<h3 class="wp-block-heading">Can this run against several servers at once?</h3>



<p class="wp-block-paragraph">Yes, but use <code>serial: 1</code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.</p>



<p class="wp-block-paragraph">Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a bootstrap playbook you can actually rerun?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:</p>



<ul class="wp-block-list">
<li>Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.</li>

<li>Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.</li>

<li>Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.</li>

<li>Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.</li>

<li>Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.</li>

<li>Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.</li>
</ul>



<p class="wp-block-paragraph">Send me the playbook, a recap from a second run, or the output of <code>ss -tlnp</code> from the host that is behaving oddly, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Automating Server Provisioning for New Clients Without Building a Snowflake Farm</title>
		<link>https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Freelancing]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=192</guid>

					<description><![CDATA[<p>Hand-built client servers fail quietly: you only find out they were never reproducible on the day you need to rebuild one. Here is how to automate server provisioning for new clients using cloud-init for first boot, Ansible for convergence, per-client secrets and a verification gate that refuses to hand over a half-built box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email is usually short. &#8220;Can you spin up a staging box that matches production?&#8221; You open your notes from eight months ago, find a half-finished shell history and a text file called <code>setup-notes-final2.txt</code>, and the honest answer is no. You can build something close. You cannot build the same machine.</p>



<p class="wp-block-paragraph">That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between &#8220;the server works&#8221; and &#8220;the server is reproducible&#8221; turns into a weekend.</p>



<p class="wp-block-paragraph">This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client&#8217;s credentials from ever touching another client&#8217;s box, and how to prove a build is finished instead of assuming it.</p>



<h2 class="wp-block-heading">The four failure families worth designing against</h2>



<p class="wp-block-paragraph">Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.</p>



<h3 class="wp-block-heading">1. You cannot rebuild it</h3>



<p class="wp-block-paragraph">The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run <code>apt install</code>, every file edited over SSH at 1am, every &#8220;I&#8217;ll document that later&#8221; is a piece of the machine that exists nowhere but the machine.</p>



<p class="wp-block-paragraph">This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.</p>



<h3 class="wp-block-heading">2. Every client is a slightly different server</h3>



<p class="wp-block-paragraph">Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.</p>



<p class="wp-block-paragraph">Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts &#8220;all our clients have&#8230;&#8221; becomes a guess.</p>



<h3 class="wp-block-heading">3. One credential, too much blast radius</h3>



<p class="wp-block-paragraph">This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client&#8217;s server, and the same API token provisions every client&#8217;s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.</p>



<p class="wp-block-paragraph">Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.</p>



<h3 class="wp-block-heading">4. The handover you cannot perform</h3>



<p class="wp-block-paragraph">Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How to automate server provisioning for new clients</h2>



<p class="wp-block-paragraph">The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.</p>



<h3 class="wp-block-heading">Layer 1: the machine itself</h3>



<p class="wp-block-paragraph">Terraform or OpenTofu talks to your provider&#8217;s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client&#8217;s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.</p>



<p class="wp-block-paragraph">Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.</p>



<h3 class="wp-block-heading">Layer 2: first boot with cloud-init</h3>



<p class="wp-block-paragraph">cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.</p>



<pre class="wp-block-code"><code>#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo</code></pre>



<p class="wp-block-paragraph">Two deliberate omissions. There is no <code>package_upgrade: true</code>, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in <code>runcmd</code> is a script you will never test again.</p>



<p class="wp-block-paragraph"><code>python3</code> is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.</p>



<p class="wp-block-paragraph">Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:</p>



<pre class="wp-block-code"><code>cloud-init schema --config-file user-data.yaml</code></pre>



<h3 class="wp-block-heading">Layer 3: convergence with Ansible</h3>



<p class="wp-block-paragraph">This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.</p>



<p class="wp-block-paragraph">The layout that scales across clients separates the shared build from the per-client differences:</p>



<pre class="wp-block-code"><code>infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml</code></pre>



<p class="wp-block-paragraph">Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.</p>



<p class="wp-block-paragraph">Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:</p>



<pre class="wp-block-code"><code>cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  &gt; ../inventory/hosts.ini</code></pre>



<p class="wp-block-paragraph">There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.</p>



<h3 class="wp-block-heading">Layer 4: the verification gate</h3>



<p class="wp-block-paragraph">This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.</p>



<p class="wp-block-paragraph">Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:</p>



<pre class="wp-block-code"><code>- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost</code></pre>



<p class="wp-block-paragraph">Note <code>ssh_unit</code> as a variable rather than a literal: the SSH unit is <code>ssh.service</code> on Debian and Ubuntu but <code>sshd.service</code> on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.</p>



<p class="wp-block-paragraph">The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.</p>



<h2 class="wp-block-heading">Keeping clients apart: secrets and access</h2>



<p class="wp-block-paragraph">Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client&#8217;s infrastructure is a single, contained action.</p>



<p class="wp-block-paragraph">For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.</p>



<pre class="wp-block-code"><code># .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...</code></pre>



<p class="wp-block-paragraph">Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.</p>



<p class="wp-block-paragraph">For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.</p>



<p class="wp-block-paragraph">If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.</p>



<h2 class="wp-block-heading">Troubleshooting a provisioning run</h2>



<h3 class="wp-block-heading">Ansible connects before cloud-init has finished</h3>



<p class="wp-block-paragraph">The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.</p>



<p class="wp-block-paragraph">Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:</p>



<pre class="wp-block-code"><code>cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log</code></pre>



<p class="wp-block-paragraph">One caveat that catches people: never put <code>cloud-init status --wait</code> inside a <code>runcmd</code> or <code>bootcmd</code> block. It will deadlock waiting for the process that is running it.</p>



<h3 class="wp-block-heading">cloud-init appears to have done nothing</h3>



<p class="wp-block-paragraph">Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in <code>/var/lib/cloud/instance/sem/</code> are preventing a rerun.</p>



<pre class="wp-block-code"><code># what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot</code></pre>



<p class="wp-block-paragraph">Never run <code>cloud-init clean</code> on a production host. It removes cached state and the following boot will behave as a first boot.</p>



<h3 class="wp-block-heading">Host key verification failed on a rebuilt server</h3>



<p class="wp-block-paragraph">Rebuild a machine on the same IP and every client with the old key in <code>known_hosts</code> refuses to connect. The tempting fix is to disable host key checking in <code>ansible.cfg</code>. Do not. That turns a real integrity check into a permanent no-op across every client you manage.</p>



<p class="wp-block-paragraph">Remove the stale entry and pin the new one as part of the provisioning flow:</p>



<pre class="wp-block-code"><code>ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 &gt;&gt; ~/.ssh/known_hosts</code></pre>



<p class="wp-block-paragraph">Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.</p>



<h3 class="wp-block-heading">The playbook reports changes on every single run</h3>



<p class="wp-block-paragraph">Almost always a <code>shell</code> or <code>command</code> task with no <code>creates</code>, <code>changed_when</code> or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.</p>



<p class="wp-block-paragraph">Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:</p>



<pre class="wp-block-code"><code>ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Putting the application build in <code>runcmd</code>. It runs once, never gets tested, and is invisible to every later change.</li>

<li>One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.</li>

<li>Copying a role into a client directory to make one small change, instead of extracting a variable.</li>

<li>Treating &#8220;the playbook exited zero&#8221; as proof the server is ready. It is proof no task threw.</li>

<li>Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.</li>

<li>Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.</li>

<li>Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Rebuild before you need to.</strong> Destroy and rebuild a client&#8217;s staging host on a schedule. It is the only honest test of reproducibility.</li>

<li><strong>Version your roles and pin them per client.</strong> A shared role that everyone tracks on main means one commit changes six clients at once.</li>

<li><strong>Keep first boot minimal.</strong> If it can wait for Ansible, it should.</li>

<li><strong>Make monitoring part of provisioning.</strong> Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.</li>

<li><strong>Same for backups.</strong> An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.</li>

<li><strong>Separate credentials per client, always.</strong> Keys, tokens, encryption keys, state backends.</li>

<li><strong>Write the runbook as you build.</strong> The provisioning repo&#8217;s README is the handover document. If it is accurate, the engagement can end cleanly.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Terraform if I only manage a few VPS instances?</h3>



<p class="wp-block-paragraph">Not necessarily. If you are creating three servers a year through a provider&#8217;s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.</p>



<h3 class="wp-block-heading">Should I build custom images instead of provisioning at boot?</h3>



<p class="wp-block-paragraph">Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.</p>



<h3 class="wp-block-heading">Terraform or OpenTofu for client projects?</h3>



<p class="wp-block-paragraph">Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.</p>



<h3 class="wp-block-heading">How do I automate server provisioning without exposing SSH to the internet?</h3>



<p class="wp-block-paragraph">The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.</p>



<h3 class="wp-block-heading">How do I stop configuration drift once the server is live?</h3>



<p class="wp-block-paragraph">Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.</p>



<h3 class="wp-block-heading">What belongs in cloud-init versus Ansible?</h3>



<p class="wp-block-paragraph">cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.</p>



<h3 class="wp-block-heading">How long should provisioning a new client server take?</h3>



<p class="wp-block-paragraph">Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer &#8220;can you rebuild this?&#8221; with yes, on any day, without opening a shell on the existing machine.</p>



<p class="wp-block-paragraph">Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.</p>



<p class="wp-block-paragraph">Start with one client. Get their staging box rebuilding from scratch. Then make the second client&#8217;s build reuse the roles from the first. That is the whole path.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a provisioning setup you can actually hand over?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:</p>



<ul class="wp-block-list">
<li>Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime</li>

<li>Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets</li>

<li>Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works</li>

<li>Writing the verification playbook that decides whether a build is finished, and wiring it into CI</li>

<li>Fixing idempotency so check-mode runs become a usable drift report instead of noise</li>

<li>Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>WP-CLI Commands That Save Hours (And the One That Bites Back)</title>
		<link>https://john-nessime.com/blog/technical-guides/wp-cli-commands-that-save-hours/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 09 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[Command Line]]></category>
		<category><![CDATA[search-replace]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[WordPress Database]]></category>
		<category><![CDATA[WordPress Maintenance]]></category>
		<category><![CDATA[WordPress Migration]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=171</guid>

					<description><![CDATA[<p>The dashboard is fine for editing a post. It is a terrible tool for anything touching a thousand rows, twelve plugins or two servers. Here are the WP-CLI commands that save hours on migrations, plugin conflicts, stalled cron and integrity checks, grouped by the job that eats the time, with the failure modes each one hides.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/wp-cli-commands-that-save-hours/">WP-CLI Commands That Save Hours (And the One That Bites Back)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;staging is approved, push it live.&#8221; Copy the files, import the database, change the two URL fields in Settings, done. The homepage loads. Then the slider is empty, a third of the images 404, and the page builder renders a grey box where a section used to be.</p>



<p class="wp-block-paragraph">That is the moment most people properly meet the command line, usually at the worst possible time. The dashboard is fine for editing a post. It is a bad tool for anything that touches a thousand rows, twelve plugins, or two servers at once.</p>



<p class="wp-block-paragraph">This is a working list of the WP-CLI commands that save hours, grouped by the job that eats the hours rather than by command name. Each section leads with the failure mode, because with a couple of these the command finishes, prints <code>Success</code>, and leaves you with a subtly broken database that nobody notices for a week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The migration hour: search-replace, and the one that bites back</h2>



<p class="wp-block-paragraph">Changing a domain by running SQL against <code>wp_posts</code> is how sites break. WordPress stores a great deal of configuration as PHP serialized arrays, and serialized strings carry their own byte length. Change <code>https://old.example.com</code> to <code>https://new.example.org</code> with a plain <code>UPDATE ... REPLACE()</code> and the length prefix no longer matches the string. PHP refuses to unserialize the value, returns <code>false</code>, and the widget or page-builder section silently renders nothing.</p>



<p class="wp-block-paragraph">The whole point of <code>wp search-replace</code> is that it unserializes, replaces, and reserializes, so the lengths stay correct. Run it in dry-run mode first, every single time:</p>



<pre class="wp-block-code"><code>wp search-replace 'https://old.example.com' 'https://new.example.org' 
  --all-tables-with-prefix 
  --skip-columns=guid 
  --precise 
  --report-changed-only 
  --dry-run</code></pre>



<p class="wp-block-paragraph">What each flag is actually doing:</p>



<ul class="wp-block-list">
<li><code>--dry-run</code> runs the whole operation and prints the report without writing. The report has a &#8220;Type&#8221; column: <code>PHP</code> means the value was serialized and handled properly, <code>SQL</code> means a plain string replacement. If you expected hundreds of changes and see zero, your search string is wrong, not the tool.</li>

<li><code>--all-tables-with-prefix</code> widens the scope to every table sharing your prefix, not just the tables registered with <code>$wpdb</code>. Plugins that create their own tables live here. <code>--all-tables</code> goes wider still and will happily rewrite tables belonging to a completely different application sharing the database, so reach for the prefix version first.</li>

<li><code>--skip-columns=guid</code> is the one people leave off. The <code>guid</code> column is a permanent identifier for feed readers, not a URL to be followed. Rewriting it makes every historical post look brand new to anything that has subscribed to your feed.</li>

<li><code>--precise</code> forces PHP-side serialization handling on every column instead of letting MySQL do the simple cases. It is slower. On a large database it is noticeably slower. It is also the difference between &#8220;probably fine&#8221; and &#8220;definitely correct&#8221;.</li>

<li><code>--report-changed-only</code> trims the report to tables that actually changed, which makes the output readable instead of a wall of zeros.</li>
</ul>



<p class="wp-block-paragraph">Read the dry-run output. When the numbers look right, run the identical command without <code>--dry-run</code>.</p>



<h3 class="wp-block-heading">The step everyone forgets</h3>



<p class="wp-block-paragraph">You changed the database. Every cache layer still holds the old values. On a site with Redis or Memcached behind a persistent object cache drop-in, the old URLs will keep being served from memory and you will spend twenty minutes convinced the replacement failed.</p>



<pre class="wp-block-code"><code>wp cache flush
wp transient delete --expired
wp rewrite flush</code></pre>



<p class="wp-block-paragraph">Be aware that on multisite with a shared persistent object cache, <code>wp cache flush</code> can clear entries for every site on the network, not just the one you are working on. Then purge the page cache and the CDN through their own controls. If you are on Cloudflare, the database change means nothing until the edge cache is purged too.</p>



<p class="wp-block-paragraph">Finally, re-run the dry-run. It should report zero remaining replacements. If it does not, you have a second URL format hiding somewhere: protocol-relative <code>//old.example.com</code>, or an escaped form like <code>https://old.example.com</code> inside JSON stored in an option. Run a second pass for each shape you find.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The plugin conflict hour: bisect without touching the dashboard</h2>



<p class="wp-block-paragraph">Classic scenario: something fatals, the admin is white, and the usual advice is &#8220;deactivate all plugins and switch to a default theme.&#8221; On a production site that means real downtime while you click through a list.</p>



<p class="wp-block-paragraph">WP-CLI has two global parameters that skip loading plugins and themes for the duration of a single command. Nothing is deactivated. Nothing is written. The site keeps serving visitors exactly as it was.</p>



<pre class="wp-block-code"><code># Does WP-CLI work at all with nothing loaded?
wp --skip-plugins --skip-themes option get siteurl

# Is it the theme?
wp --skip-themes option get siteurl

# Skip one specific plugin
wp --skip-plugins=akismet option get siteurl</code></pre>



<p class="wp-block-paragraph">If the first command works and the unmodified one does not, the fault is in your plugins or theme, not in WP-CLI or WordPress core. From there you can walk the active list one at a time. This one-liner from the WP-CLI handbook does the walk for you:</p>



<pre class="wp-block-code"><code>wp plugin list --field=name --status=active --skip-plugins 
  | xargs -n1 -I % wp --skip-plugins=% plugin get % --field=name</code></pre>



<p class="wp-block-paragraph">The inner <code>--skip-plugins</code> on the listing command matters. Without it, a plugin broken badly enough to crash the bootstrap will also crash the command that is trying to build the list, and you get nothing.</p>



<p class="wp-block-paragraph">The honest limitation: skipping a plugin at the CLI does not reproduce browser-side behaviour. If the bug only appears for logged-in users on a specific admin screen, this narrows the field but will not fully confirm the culprit. It still turns a thirty-minute click-through into about two minutes of scrolling output.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The silent-cron hour: scheduled jobs that quietly stopped</h2>



<p class="wp-block-paragraph">This is the invisible failure. Nobody opens a ticket saying &#8220;cron is broken.&#8221; They open a ticket saying backups have not run since last month, or scheduled posts are stuck in the future, or order emails stopped going out. WP-Cron is not a scheduler. It is a queue that gets drained when someone loads a page, which means a low-traffic site drains it rarely and a fully page-cached site may never drain it at all.</p>



<pre class="wp-block-code"><code># Can the site reach its own wp-cron.php?
wp cron test

# What is scheduled, and how overdue is it?
wp cron event list

# Drain everything that is due, right now, in this process
wp cron event run --due-now</code></pre>



<p class="wp-block-paragraph">In <code>wp cron event list</code>, the column to read is <code>next_run_relative</code>. If events are showing as long overdue, the queue is not being drained and you have found your problem.</p>



<p class="wp-block-paragraph">The fix is to stop relying on page loads. Disable the loopback trigger in <code>wp-config.php</code>, above the &#8220;stop editing&#8221; line:</p>



<pre class="wp-block-code"><code>define( 'DISABLE_WP_CRON', true );</code></pre>



<p class="wp-block-paragraph">Then add a real system cron entry for the user that owns the site files:</p>



<pre class="wp-block-code"><code>* * * * * /usr/local/bin/wp --path=/var/www/example.com cron event run --due-now --quiet</code></pre>



<p class="wp-block-paragraph">Why the CLI form and not a <code>curl</code> of <code>wp-cron.php</code>: <code>wp cron event run</code> executes the events in the current PHP process. No HTTP loopback is involved, so it cannot be defeated by a firewall rule, a reverse proxy, HTTP authentication on staging, or a host that rate-limits self-requests. Errors surface in the cron mail or the log you redirect to, instead of vanishing into a request nobody reads.</p>



<p class="wp-block-paragraph">One warning. Set <code>DISABLE_WP_CRON</code> and then forget the crontab entry, and you have upgraded an unreliable scheduler into one that never runs at all. Verify with <code>wp cron event list</code> a few minutes later before you close the ticket.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The bulk-edit hour: piping IDs instead of clicking pages</h2>



<p class="wp-block-paragraph">The pattern that does the real work here is <code>--format=ids</code> feeding another command. Once that clicks, most &#8220;there is no plugin for this&#8221; tasks become one line.</p>



<pre class="wp-block-code"><code># How many revisions are we carrying?
wp post list --post_type=revision --format=count

# Delete them, skipping the trash
wp post delete $(wp post list --post_type=revision --format=ids) --force

# Every administrator on the site, in one table
wp user list --role=administrator --fields=ID,user_login,user_email

# Regenerate only the thumbnail sizes that are missing
wp media regenerate --only-missing --yes</code></pre>



<p class="wp-block-paragraph">Two things to know before you run the delete. It is irreversible, so take a database export first. And if the inner command returns nothing because there are no revisions, the outer command gets no arguments and will error rather than doing nothing quietly. That is a harmless failure, but it looks alarming the first time.</p>



<p class="wp-block-paragraph">Revisions come back unless you cap them. Adding <code>define( 'WP_POST_REVISIONS', 5 );</code> to <code>wp-config.php</code> keeps enough history to recover from a bad edit without letting <code>wp_posts</code> grow without limit.</p>



<p class="wp-block-paragraph">For anything you want to script rather than read, add <code>--format=json</code> and pipe it into <code>jq</code>. Every list command in WP-CLI supports it, which makes the whole tool usable as a data source for monitoring and reporting rather than just an admin replacement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The &#8220;is this thing compromised?&#8221; hour: checksums</h2>



<p class="wp-block-paragraph">A site is behaving oddly, there is a suspicious redirect, and the first instinct is to install a security scanner. Before that, run the two commands that compare what is on disk against the official hashes on WordPress.org.</p>



<pre class="wp-block-code"><code>wp core verify-checksums
wp plugin verify-checksums --all</code></pre>



<p class="wp-block-paragraph">These download the published checksums for your exact version and locale and report every core or plugin file that has been modified, plus files that should not exist at all. The core command deliberately avoids loading WordPress, so it still works when the site itself is too broken to boot.</p>



<p class="wp-block-paragraph">Know the blind spots before you trust a clean result:</p>



<ul class="wp-block-list">
<li>Only plugins hosted on the WordPress.org directory can be verified. Premium plugins, custom code and anything from a private repository have no published source of truth and will be skipped or warned about.</li>

<li>Nothing in <code>wp-content/uploads</code> is covered. A dropped PHP file in an uploads subdirectory is a common backdoor and checksums will never see it.</li>

<li>Themes are not covered by an equivalent bundled command.</li>

<li>Getting a locale or version mismatch produces alarming warnings that are not real findings. Pass <code>--version</code> and <code>--locale</code> explicitly if the output looks wrong.</li>
</ul>



<p class="wp-block-paragraph">A clean checksum run is not proof the site is clean. A dirty one is proof it is not, which is worth a great deal in the first five minutes of an incident.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The database hour: find the bloat before you optimise anything</h2>



<p class="wp-block-paragraph">&#8220;The site is slow&#8221; is not actionable. &#8220;One table is carrying most of the database&#8221; is. Start by looking, not by running an optimiser.</p>



<pre class="wp-block-code"><code># Size per table, sorted by whatever is worst
wp db size --tables --human-readable

# Take an export before touching anything
wp db export ~/backups/example-com.sql

# Run a query and strip the ASCII table borders
wp db query 'SELECT option_value FROM wp_options WHERE option_name="home"' --skip-column-names</code></pre>



<p class="wp-block-paragraph">In practice the offenders are nearly always <code>wp_options</code> stuffed with autoloaded rows and orphaned transients, <code>wp_postmeta</code> from a plugin that never cleans up after itself, or <code>wp_posts</code> full of revisions. Autoloaded options are the sneaky one, because every row marked for autoload is loaded on every request, including requests that will never look at it.</p>



<p class="wp-block-paragraph">Two clean-up commands worth knowing, and the difference between them:</p>



<pre class="wp-block-code"><code>wp transient delete --expired
wp transient delete --all</code></pre>



<p class="wp-block-paragraph">The first removes only what has already expired and is safe to run on a schedule. The second removes everything, including live cache entries, which forces WordPress to rebuild them on the next requests. That is safe but produces a short performance dip, so it belongs in a maintenance window rather than in a cron job.</p>



<p class="wp-block-paragraph">Note that on a site with a persistent object cache drop-in installed, transients bypass the database entirely and wrap the object cache instead. If <code>wp transient delete --all</code> reports nothing, that is usually the reason and not a failure.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The setup that makes everything above faster</h2>



<p class="wp-block-paragraph">If you manage more than one site, the biggest single time saving is not a command at all. It is aliases. Put a <code>wp-cli.yml</code> in a project directory, or <code>~/.wp-cli/config.yml</code> for a global one:</p>



<pre class="wp-block-code"><code>@production:
  ssh: deploy@example.com/var/www/example.com
@staging:
  ssh: deploy@staging.example.com/var/www/staging</code></pre>



<p class="wp-block-paragraph">Now you run commands against a remote site from your own machine, without an SSH session:</p>



<pre class="wp-block-code"><code>wp @staging plugin list --status=active
wp @production cron event list</code></pre>



<p class="wp-block-paragraph">This needs WP-CLI installed on the remote host, and it needs your host to actually give you SSH. That varies more than people expect. A plain VPS from a provider like InterServer or DigitalOcean gives you root and you install what you like. Managed WordPress hosts differ: some ship WP-CLI preinstalled and hand you an SSH key immediately, others restrict which commands you can run, and a few give you no shell at all on entry-level plans. Worth checking before you build a workflow that depends on it.</p>



<p class="wp-block-paragraph">Two optional packages are worth the install if you do maintenance work regularly:</p>



<pre class="wp-block-code"><code>wp package install wp-cli/doctor-command
wp package install wp-cli/profile-command

wp doctor check --all
wp profile stage</code></pre>



<p class="wp-block-paragraph"><code>wp doctor</code> runs a set of health checks, including one for autoloaded options size, and gives you a pass or warn per check. <code>wp profile</code> breaks a request into stages so you can see where the time is going before you start guessing.</p>



<p class="wp-block-paragraph">The honest trade-off: these are separate packages, not bundled commands, and they get less attention than the core set. If you are working across client servers you do not control, installing packages everywhere is friction you may not want. On your own infrastructure they earn their place. For continuous visibility rather than point-in-time checks you still want proper monitoring, whether that is a self-hosted Prometheus and Grafana stack or a hosted uptime service.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: when WP-CLI itself will not run</h2>



<p class="wp-block-paragraph">Before blaming the tool, run <code>wp cli info</code>. It prints the PHP binary in use, the PHP version, the loaded <code>php.ini</code>, and the WP-CLI version. Most problems are visible in that output.</p>



<ul class="wp-block-list">
<li><strong>&#8220;This does not seem to be a WordPress installation.&#8221;</strong> You are in the wrong directory, or the document root is elsewhere. Pass <code>--path=/full/path/to/wordpress</code> rather than guessing.</li>

<li><strong>Wrong PHP version.</strong> On a control panel server the shell PHP is often not the PHP the site runs on. Check the binary in <code>wp cli info</code>, and invoke the right one explicitly if they differ, for example <code>/usr/local/php83/bin/php $(which wp)</code>.</li>

<li><strong>Memory exhaustion on large operations.</strong> The CLI uses its own memory limit, not the web one. Raise it for a single command with <code>php -d memory_limit=512M $(which wp) ...</code> instead of editing configuration globally.</li>

<li><strong>A fatal error the moment you type anything.</strong> Confirm with <code>wp --skip-plugins --skip-themes cli info</code>. If that works, the fault is in site code.</li>

<li><strong>A warning about running as root.</strong> WP-CLI is telling you that files created by this command will be owned by root and the web server may not be able to write to them later. <code>--allow-root</code> silences the warning without fixing the ownership problem. Run as the site user instead, unless you are inside a container where root is the normal case.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running <code>search-replace</code> without <code>--dry-run</code> because you have done it before and it worked.</li>

<li>Reaching for <code>--all-tables</code> by reflex when <code>--all-tables-with-prefix</code> is what you meant, on a server where several sites share one database.</li>

<li>Forgetting to flush the object cache and CDN after a database write, then re-running the replacement several times chasing a ghost.</li>

<li>Setting <code>DISABLE_WP_CRON</code> without adding the system cron entry.</li>

<li>Treating a clean <code>verify-checksums</code> result as proof that a site is not compromised.</li>

<li>Running destructive commands over SSH without <code>screen</code> or <code>tmux</code>, then losing the connection mid-write on a large table.</li>

<li>Using <code>--allow-root</code> as a habit and leaving root-owned files scattered through <code>wp-content</code>.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ol class="wp-block-list">
<li>Export the database before any command that writes. <code>wp db export</code> takes seconds and has saved more afternoons than any other command in this post.</li>

<li>Dry-run first, read the report, then repeat the command verbatim without the flag. Do not retype it.</li>

<li>Run as the site user, not root, so file ownership stays correct.</li>

<li>Use aliases in <code>wp-cli.yml</code> so you cannot run a staging command against production by muscle memory.</li>

<li>Verify after the write. Re-run the read-only version of whatever you just did and confirm it reports zero remaining work.</li>

<li>Test the restore path, not just the backup. An export you have never imported is a hope, not a backup.</li>

<li>Keep WP-CLI current with <code>wp cli update</code>, particularly before working on a site running a recent PHP release.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is wp search-replace safe to run on a live site?</h3>



<p class="wp-block-paragraph">Safer than raw SQL, because it handles serialized data correctly. Still a write against production. Export the database first, dry-run first, and use <code>--skip-columns=guid</code>. On a large database the operation can hold the site in a slightly inconsistent state for the duration, so a quiet window helps.</p>



<h3 class="wp-block-heading">Why do my changes not show up after search-replace?</h3>



<p class="wp-block-paragraph">Almost always caching. Run <code>wp cache flush</code>, purge the page cache and the CDN, then reload. If the old value still appears in a dry-run report, you have a second URL format stored somewhere, such as a protocol-relative or escaped variant.</p>



<h3 class="wp-block-heading">Does WP-CLI need SSH access?</h3>



<p class="wp-block-paragraph">It needs shell access on a machine that can reach the WordPress files and database. That usually means SSH. Some managed hosts provide a browser terminal instead, and some restrict the available commands. On a self-managed VPS you install it yourself and nothing is restricted.</p>



<h3 class="wp-block-heading">What is the difference between &#8211;skip-plugins and deactivating a plugin?</h3>



<p class="wp-block-paragraph"><code>--skip-plugins</code> prevents the plugin from loading for that one command only. Nothing is written to the database and visitors are unaffected. Deactivating changes site state and takes the plugin offline for everyone, which is why the skip flags are the right diagnostic tool on production.</p>



<h3 class="wp-block-heading">Can I schedule WP-CLI commands with cron?</h3>



<p class="wp-block-paragraph">Yes, and that is the recommended way to run WordPress scheduled events reliably. Use the full path to the <code>wp</code> binary, pass <code>--path</code>, run as the site user, and redirect output so failures are visible rather than silent.</p>



<h3 class="wp-block-heading">Will verify-checksums detect all malware?</h3>



<p class="wp-block-paragraph">No. It compares core files and WordPress.org-hosted plugins against published hashes. It does not cover themes, premium or custom plugins, or anything in the uploads directory. Treat a failure as a definite finding and a pass as one data point among several.</p>



<h3 class="wp-block-heading">Which WP-CLI commands should I learn first?</h3>



<p class="wp-block-paragraph"><code>wp db export</code>, <code>wp search-replace --dry-run</code>, <code>wp --skip-plugins</code>, and <code>wp cron event list</code>. Those four cover backup, migration, conflict isolation and the most common silent failure on a WordPress site.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most of the WP-CLI commands that save hours do it by turning a job you cannot see into a job you can read. <code>--dry-run</code> shows you the write before it happens. <code>wp cron event list</code> shows you a queue that was failing silently. <code>verify-checksums</code> shows you a file that changed without anyone deciding it should.</p>



<p class="wp-block-paragraph">The speed is real, but the speed is not the point. The point is that you stop guessing. If you take one habit from this post, take the dry-run: run the read-only version, read the output, then run the identical command for real. That single pattern prevents the majority of the incidents I see caused by command-line work on WordPress.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with this?</h2>



<p class="wp-block-paragraph">I work on WordPress infrastructure, and command-line automation is most of what makes that maintainable. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Domain and host migrations run properly, with staged dry-runs, correct table scoping and a verified rollback before anything is written.</li>

<li>Replacing WP-Cron with real system cron across a fleet, including alerting when a job stops firing instead of finding out weeks later.</li>

<li>Writing repeatable WP-CLI maintenance scripts, with proper exit codes and logging, so they can be scheduled rather than remembered.</li>

<li>Diagnosing slow WordPress sites at the database layer, from autoloaded options and table bloat down to the query causing it.</li>

<li>Incident triage on a site suspected of being compromised, starting with integrity checks and file-level evidence rather than a scanner plugin.</li>

<li>Setting up multi-site management workflows with WP-CLI aliases, SSH keys and deployment pipelines that do not depend on anyone opening the dashboard.</li>
</ul>



<p class="wp-block-paragraph">If something specific is going wrong, send the actual output. A dry-run report, a <code>wp cli info</code> dump, a <code>wp cron event list</code> table, and I can usually tell you where the problem is before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/wp-cli-commands-that-save-hours/">WP-CLI Commands That Save Hours (And the One That Bites Back)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</title>
		<link>https://john-nessime.com/blog/system-administration/vps-hardening-checklist/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:24 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=62</guid>

					<description><![CDATA[<p>You set PasswordAuthentication no, restarted SSH, and moved on. Months later the logs show successful password logins, because a vendor drop-in file sorted earlier and won. A VPS setup and hardening checklist where every step comes with the command that proves it worked.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You spun up a new box, worked through a hardening guide, set <code>PasswordAuthentication no</code>, restarted SSH, and moved on. Six months later you are reading auth logs for an unrelated reason and there it is: successful password logins. Not attempts. Logins.</p>



<p class="wp-block-paragraph">The line you wrote was real. The file was saved. What you did not know is that your distro puts an <code>Include /etc/ssh/sshd_config.d/*.conf</code> near the top of <code>sshd_config</code>, that sshd uses the <em>first</em> value it finds for any given keyword, and that your cloud provider&#8217;s image dropped a <code>50-cloud-init.conf</code> in that directory setting <code>PasswordAuthentication yes</code>. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.</p>



<p class="wp-block-paragraph">You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.</p>



<p class="wp-block-paragraph">That is the thing most guides get wrong, and it is the idea this <strong>VPS hardening checklist</strong> is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.</p>



<p class="wp-block-paragraph">This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.</p>



<h2 class="wp-block-heading">Before you touch anything</h2>



<p class="wp-block-paragraph">Two minutes of preparation that saves an evening.</p>



<p class="wp-block-paragraph"><strong>Find your provider&#8217;s console before you need it.</strong> Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.</p>



<p class="wp-block-paragraph"><strong>Never close the session you are working in.</strong> Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.</p>



<p class="wp-block-paragraph">Then update the system, because everything else assumes current packages:</p>



<pre class="wp-block-code"><code>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo reboot   # if a kernel update landed</code></pre>



<h2 class="wp-block-heading">1. A real user, and keys instead of passwords</h2>



<p class="wp-block-paragraph">Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.</p>



<pre class="wp-block-code"><code># On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"</code></pre>



<pre class="wp-block-code"><code># On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10</code></pre>



<p class="wp-block-paragraph">Now open a second terminal and confirm <code>ssh deploy@203.0.113.10</code> works, and that <code>sudo -v</code> works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.</p>



<h2 class="wp-block-heading">2. SSH config, and why your edit might do nothing</h2>



<p class="wp-block-paragraph">This is the section that actually matters. Two mechanisms silently override what you write.</p>



<h3 class="wp-block-heading">Drop-in files and first-match-wins</h3>



<p class="wp-block-paragraph">Modern OpenSSH packages ship an <code>Include</code> line near the top of <code>/etc/ssh/sshd_config</code>, which pulls in every <code>.conf</code> file from <code>/etc/ssh/sshd_config.d/</code> in alphabetical order, before the rest of the main file is read. Combine that with sshd&#8217;s rule that the first value found for a keyword wins, and the consequence is specific: <strong>your drop-in has to sort earlier than the vendor&#8217;s, not later.</strong></p>



<p class="wp-block-paragraph">Most guides tell you to name it <code>99-something.conf</code>, which is the convention almost everywhere else in Linux and exactly backwards here.</p>



<pre class="wp-block-code"><code># /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy</code></pre>



<p class="wp-block-paragraph">A note on <code>KbdInteractiveAuthentication</code>: older guides and older OpenSSH call this <code>ChallengeResponseAuthentication</code>. Setting only <code>PasswordAuthentication no</code> can still leave a keyboard-interactive path open on some configurations, which is why both are here.</p>



<p class="wp-block-paragraph">Now the part nobody does. Validate the syntax, then read the effective config:</p>



<pre class="wp-block-code"><code># Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'</code></pre>



<p class="wp-block-paragraph">If <code>passwordauthentication yes</code> comes back, your file lost to something else in that directory. Look at what is in there with <code>ls /etc/ssh/sshd_config.d/</code> and either rename yours to sort earlier or edit the offending file directly.</p>



<h3 class="wp-block-heading">Socket activation, and the port that will not change</h3>



<p class="wp-block-paragraph">The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: <code>ssh.socket</code> holds the listening port and starts <code>sshd</code> only when a connection arrives. On those systems the <code>Port</code> and <code>ListenAddress</code> directives in <code>sshd_config</code> may be ignored entirely, because systemd owns the socket.</p>



<p class="wp-block-paragraph">People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:</p>



<pre class="wp-block-code"><code># -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp</code></pre>



<p class="wp-block-paragraph">If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under <code>/lib/systemd/system/</code>, which a package update will overwrite:</p>



<pre class="wp-block-code"><code>sudo systemctl edit ssh.socket</code></pre>



<pre class="wp-block-code"><code># In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222</code></pre>



<pre class="wp-block-code"><code>sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it</code></pre>



<p class="wp-block-paragraph">Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">3. Firewall, in the right order</h2>



<p class="wp-block-paragraph">The ordering here is the whole trick. Set the default policy and add your rules <em>before</em> you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.</p>



<pre class="wp-block-code"><code>sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose</code></pre>



<p class="wp-block-paragraph">Read that <code>status verbose</code> output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.</p>



<p class="wp-block-paragraph">One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with <code>-p 5432:5432</code> is reachable from the internet even though ufw says that port is denied. Bind container ports to <code>127.0.0.1</code> explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.</p>



<p class="wp-block-paragraph">A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.</p>



<h2 class="wp-block-heading">4. Automatic security updates</h2>



<p class="wp-block-paragraph">Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.</p>



<pre class="wp-block-code"><code>sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug</code></pre>



<p class="wp-block-paragraph">By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.</p>



<h2 class="wp-block-heading">5. fail2ban, and what it is actually for</h2>



<p class="wp-block-paragraph">Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.</p>



<p class="wp-block-paragraph">What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials <em>do</em> exist: a mail server, a web application login form, an FTP daemon.</p>



<pre class="wp-block-code"><code>sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd</code></pre>



<p class="wp-block-paragraph">Check that <code>status</code> output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, <code>journalctl -u fail2ban</code> will tell you why.</p>



<p class="wp-block-paragraph">Set <code>ignoreip</code> in <code>jail.local</code> to include your office or home address. Banning yourself is a rite of passage but it is avoidable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">6. The boring items that matter more than the exciting ones</h2>



<p class="wp-block-paragraph">Nobody writes blog posts about these, and they cause more real outages than any of the above.</p>



<h3 class="wp-block-heading">Provider account security</h3>



<p class="wp-block-paragraph">Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.</p>



<h3 class="wp-block-heading">Time and timezone</h3>



<pre class="wp-block-code"><code>sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"</code></pre>



<p class="wp-block-paragraph">UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.</p>



<h3 class="wp-block-heading">Swap on small instances</h3>



<pre class="wp-block-code"><code># A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h</code></pre>



<h3 class="wp-block-heading">Backups that leave the machine</h3>



<p class="wp-block-paragraph">Your provider&#8217;s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.</p>



<p class="wp-block-paragraph">Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man&#8217;s switch service at the backup job so you find out when it stops running, rather than the day you need it.</p>



<h3 class="wp-block-heading">Knowing the box is alive</h3>



<p class="wp-block-paragraph">At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.</p>



<h2 class="wp-block-heading">What I would skip</h2>



<p class="wp-block-paragraph">Not everything in the long hardening guides earns its cost on a single web server.</p>



<ul class="wp-block-list">
<li><strong>Large sysctl hardening blocks pasted from a gist.</strong> Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.</li>
<li><strong>AIDE, auditd and file-integrity monitoring</strong> on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.</li>
<li><strong>Rebuilding OpenSSH&#8217;s cipher list.</strong> Current defaults are good. Guides that hand you a <code>Ciphers</code> line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.</li>
<li><strong>Disabling ping.</strong> It breaks your own monitoring and hides nothing.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Locked out after an SSH change</h3>



<p class="wp-block-paragraph">Provider console, log in as root, and revert. If you cannot remember what you changed, <code>ls -lt /etc/ssh/sshd_config.d/</code> shows the most recently modified file. Run <code>sshd -t</code> to see whether the config is even valid, since a syntax error stops sshd starting at all.</p>



<h3 class="wp-block-heading">Key auth rejected, password prompt appears instead</h3>



<p class="wp-block-paragraph">Almost always permissions. The home directory must not be group-writable, <code>~/.ssh</code> needs 700 and <code>authorized_keys</code> 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run <code>ssh -v</code> to see how far it gets, and check <code>journalctl -u ssh</code> on the server for the actual reason.</p>



<h3 class="wp-block-heading">The setting I applied is not in effect</h3>



<p class="wp-block-paragraph"><code>sudo sshd -T</code> and search for the keyword. If the effective value differs from your file, something in <code>sshd_config.d/</code> sorts earlier, or a <code>Match</code> block later in the config is overriding it for your connection.</p>



<h3 class="wp-block-heading">A service is reachable that the firewall says is blocked</h3>



<p class="wp-block-paragraph">Docker, or another daemon writing netfilter rules directly. Check with <code>sudo iptables -S</code> or <code>sudo nft list ruleset</code>, and always verify from a machine outside the server rather than trusting <code>ufw status</code>.</p>



<h3 class="wp-block-heading">Nothing obviously wrong, but who has been logging in?</h3>



<pre class="wp-block-code"><code>last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Closing your only SSH session before testing the new configuration from a second one.</li>
<li>Enabling the firewall before adding the SSH rule.</li>
<li>Naming your sshd drop-in <code>99-</code>, so a vendor file that sorts earlier wins.</li>
<li>Trusting the file you edited instead of running <code>sshd -T</code>.</li>
<li>Generating the SSH key on the server.</li>
<li>Changing the SSH port and counting it as a security control.</li>
<li>Disabling password auth for SSH while leaving a web control panel with a weak password exposed.</li>
<li>No two-factor authentication on the hosting account that can reset root.</li>
<li>Treating provider snapshots as backups.</li>
<li>Installing fail2ban and never checking that it started.</li>
<li>Publishing container ports without binding them to localhost, then assuming ufw covers it.</li>
<li>Pasting a sysctl block you cannot explain line by line.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Verify every control after applying it, with a command that reads the running state.</li>
<li>Keep a second session open and the provider console tested before making SSH changes.</li>
<li>Key-only authentication, root login disabled, and an explicit <code>AllowUsers</code> list.</li>
<li>Default-deny inbound firewall, with rules added before the firewall is enabled.</li>
<li>Automatic security updates on, with a deliberate decision about automatic reboots.</li>
<li>Two-factor authentication on the hosting account and the registrar.</li>
<li>Backups off the machine, on separate credentials, and restore-tested at least once.</li>
<li>UTC everywhere, with clock sync confirmed.</li>
<li>Monitoring that pages you on disk usage before it pages you on downtime.</li>
<li>Script the whole build, so the next server is identical and you are not repeating this from memory.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the single most important step?</h3>



<p class="wp-block-paragraph">Key-only SSH with root login disabled, verified with <code>sshd -T</code>. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.</p>



<h3 class="wp-block-heading">Should I change the SSH port?</h3>



<p class="wp-block-paragraph">For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.</p>



<h3 class="wp-block-heading">Do I need fail2ban if password authentication is off?</h3>



<p class="wp-block-paragraph">Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.</p>



<h3 class="wp-block-heading">Why did my sshd_config change get ignored?</h3>



<p class="wp-block-paragraph">Two likely causes. A file in <code>/etc/ssh/sshd_config.d/</code> that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is <code>Port</code> or <code>ListenAddress</code> on a socket-activated system, where systemd owns the listener. <code>sshd -T</code> answers the first case and <code>ss -tlnp</code> answers the second.</p>



<h3 class="wp-block-heading">Is ufw enough, or should I use nftables directly?</h3>



<p class="wp-block-paragraph">ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.</p>



<h3 class="wp-block-heading">How do I harden a server I did not build?</h3>



<p class="wp-block-paragraph">Audit before you change anything. <code>sshd -T</code>, <code>ufw status verbose</code>, <code>ss -tlnp</code> for unexpected listeners, <code>systemctl list-units --type=service --state=running</code>, and the contents of every <code>authorized_keys</code> on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.</p>



<h3 class="wp-block-heading">Should this be automated?</h3>



<p class="wp-block-paragraph">Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Editing a config file is not the same as changing a system&#8217;s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.</p>



<p class="wp-block-paragraph">So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. <code>sshd -T</code> for SSH, <code>ufw status verbose</code> for the firewall, <code>ss -tlnp</code> for what is listening, <code>systemctl status</code> for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.</p>



<h2 class="wp-block-heading">Want a second pair of eyes on your server?</h2>



<p class="wp-block-paragraph">Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:</p>



<ul class="wp-block-list">
<li>Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.</li>
<li>Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.</li>
<li>Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.</li>
<li>Moving SSH off the public internet onto a private overlay network or bastion.</li>
<li>Backup strategy that survives the hosting account being compromised, including a tested restore.</li>
<li>Firewall and container networking, especially where published Docker ports are bypassing the firewall.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>sshd -T</code>, <code>ufw status verbose</code> and <code>ss -tlnp</code>, and I will tell you what stands out.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</title>
		<link>https://john-nessime.com/blog/devops/github-actions-vps-deployment/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:21 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=66</guid>

					<description><![CDATA[<p>A deploy key in GitHub secrets is a shell on your production server, handed to a container you don't control, running code from maintainers you've never met. Here's how to build the pipeline so a leaked key isn't worth much: forced commands, scoped sudo, pinned actions and a real approval gate.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Stop and look at what you are about to build. A private SSH key that can log into your production server, pasted into a text box on github.com, handed to a container you do not control, which then runs code from repositories maintained by people you have never met.</p>



<p class="wp-block-paragraph">That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer&#8217;s token, pushed a malicious commit, and then <em>retroactively repointed every version tag</em> at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner&#8217;s secrets into the build log. On public repositories, those logs are readable by anyone.</p>



<p class="wp-block-paragraph">Nobody&#8217;s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.</p>



<p class="wp-block-paragraph">A <strong>GitHub Actions VPS deployment</strong> is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.</p>



<h2 class="wp-block-heading">Who can actually reach that key</h2>



<p class="wp-block-paragraph">Worth being explicit, because it is a longer list than people expect:</p>



<ul class="wp-block-list">
<li><strong>Anyone with write access to the repository.</strong> A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.</li>
<li><strong>Every action in the job.</strong> Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.</li>
<li><strong>Anyone who can influence what runs.</strong> This is why <code>pull_request_target</code> combined with checking out the pull request&#8217;s own code is such a well-known foot-gun: it runs the fork&#8217;s code with the base repository&#8217;s secrets in scope.</li>
</ul>



<p class="wp-block-paragraph">The goal is not to make the key unreachable. It is to make a stolen key not worth much.</p>



<h2 class="wp-block-heading">Decide how the runner authenticates</h2>



<p class="wp-block-paragraph">Three options, and the right one depends on what you already run.</p>



<ol class="wp-block-list">
<li><strong>A dedicated SSH key in repository secrets.</strong> The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.</li>
<li><strong>An overlay network.</strong> Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.</li>
<li><strong>A self-hosted runner on the server.</strong> No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.</li>
</ol>



<p class="wp-block-paragraph">One thing that sounds appealing and is not practical: allowlisting GitHub&#8217;s runner IP ranges at your firewall. The ranges are published through GitHub&#8217;s meta API, but they are enormous and they change, and every other GitHub customer&#8217;s job runs from inside them. It buys you very little for a lot of maintenance.</p>



<h2 class="wp-block-heading">Narrow the key until it is boring</h2>



<p class="wp-block-paragraph">Assume the key leaks. What can the holder do? Work backwards from there.</p>



<p class="wp-block-paragraph"><strong>Give it its own user.</strong> A <code>deploy</code> account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.</p>



<p class="wp-block-paragraph"><strong>Restrict the key in <code>authorized_keys</code>.</strong> SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:</p>



<pre class="wp-block-code"><code># ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions</code></pre>



<p class="wp-block-paragraph">The guard script decides what the key is allowed to do. Keep it short enough to read in one go:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" &gt;&amp;2
    exit 1
    ;;
esac</code></pre>



<p class="wp-block-paragraph">Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.</p>



<p class="wp-block-paragraph"><strong>Scope the sudo entry too.</strong> The deploy user needs to reload a service, not run arbitrary commands as root:</p>



<pre class="wp-block-code"><code># visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx</code></pre>



<p class="wp-block-paragraph">Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted <code>deploy</code> user with no sudo is a reasonable stopping point.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Verify the server, not just the key</h2>



<p class="wp-block-paragraph">Almost every tutorial on this topic contains <code>StrictHostKeyChecking=no</code>, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.</p>



<pre class="wp-block-code"><code># Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com</code></pre>



<p class="wp-block-paragraph">Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.</p>



<h2 class="wp-block-heading">The workflow</h2>



<pre class="wp-block-code"><code>name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" &gt; ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" &gt; ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz &gt; /dev/null</code></pre>



<p class="wp-block-paragraph">Three details in there worth calling out, because they are the ones people leave out.</p>



<p class="wp-block-paragraph"><code>-o BatchMode=yes</code> stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.</p>



<p class="wp-block-paragraph"><code>--delete</code> is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if <code>.deployignore</code> is wrong. Run it once with <code>--dry-run</code> against staging and read the list before you point it at production.</p>



<p class="wp-block-paragraph">The final <code>curl --fail</code> is the difference between &#8220;the files copied&#8221; and &#8220;the site works&#8221;. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.</p>



<h2 class="wp-block-heading">Third-party actions</h2>



<p class="wp-block-paragraph">There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain <code>ssh</code> and <code>rsync</code> for that reason.</p>



<p class="wp-block-paragraph">Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.</p>



<p class="wp-block-paragraph">Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.</p>



<h2 class="wp-block-heading">Gate production properly</h2>



<p class="wp-block-paragraph">The <code>environment:</code> key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:</p>



<ul class="wp-block-list">
<li><strong>Scoped secrets.</strong> The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.</li>
<li><strong>Required reviewers.</strong> The job pauses and waits for a named human to approve it. This is the single control that turns &#8220;anyone with write access can deploy&#8221; into &#8220;anyone with write access can request a deploy&#8221;.</li>
<li><strong>Deployment branch and tag rules.</strong> Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.</li>
</ul>



<p class="wp-block-paragraph">Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.</p>



<p class="wp-block-paragraph">For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Permission denied (publickey)</h3>



<p class="wp-block-paragraph">The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses <code>printf '%sn'</code> rather than <code>echo</code>: it guarantees the newline regardless of what got stored.</p>



<p class="wp-block-paragraph">After that, check permissions on the server: <code>~/.ssh</code> must be 700 and <code>authorized_keys</code> 600, owned by the deploy user, and the home directory must not be group-writable. Add <code>-vvv</code> to the ssh command to see how far the handshake gets, and read <code>journalctl -u ssh</code> on the server for the reason, since the client is deliberately vague.</p>



<h3 class="wp-block-heading">Host key verification failed</h3>



<p class="wp-block-paragraph">Either the <code>known_hosts</code> variable is empty, or the server&#8217;s host key genuinely changed. If you rebuilt the server, regenerate it with <code>ssh-keyscan</code> and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.</p>



<h3 class="wp-block-heading">The job hangs until the timeout</h3>



<p class="wp-block-paragraph">SSH is waiting on a prompt nobody will answer. Add <code>-o BatchMode=yes</code> and it fails immediately with a readable reason instead.</p>



<h3 class="wp-block-heading">rsync works from my laptop, fails from the runner</h3>



<p class="wp-block-paragraph">If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records <code>$SSH_ORIGINAL_COMMAND</code>, run the job once, and read what rsync actually sent.</p>



<h3 class="wp-block-heading">Works on push, fails on pull requests</h3>



<p class="wp-block-paragraph">Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for <code>pull_request_target</code> to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.</p>



<h3 class="wp-block-heading">A secret appeared in the logs</h3>



<p class="wp-block-paragraph">Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using an existing personal SSH key instead of generating a dedicated one for the runner.</li>
<li>Giving the deploy user full sudo, or a sudo rule with a wildcard in it.</li>
<li>Disabling host key checking with <code>StrictHostKeyChecking=no</code>.</li>
<li>Referencing third-party actions by tag in the job that holds your production key.</li>
<li>Interpolating <code>${{ secrets.X }}</code> directly into a <code>run:</code> script instead of passing it through <code>env:</code>.</li>
<li>Leaving default write permissions on a job that only needs to read.</li>
<li>Using <code>cancel-in-progress: true</code> on a deploy, so a second push interrupts the first mid-copy.</li>
<li>Running <code>rsync --delete</code> at production without testing the exclude list.</li>
<li>Treating &#8220;the files copied&#8221; as a successful deploy, with no health check.</li>
<li>Deploying straight from branch pushes with no approval step.</li>
<li>Relying on log masking to keep a leaked secret out of the output.</li>
<li>Never rotating the deploy key, including after someone leaves the team.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Dedicated deploy user, dedicated key, used for nothing else.</li>
<li>Restrict the key in <code>authorized_keys</code> with <code>restrict</code> and a forced command.</li>
<li>Scope sudo to the exact commands the deploy needs.</li>
<li>Pin every action to a full commit SHA and let Dependabot bump them.</li>
<li>Keep the deploy job small: fewer steps means fewer things that can read the secret.</li>
<li>Set <code>permissions:</code> explicitly, defaulting to read-only.</li>
<li>Verify host keys from a stored <code>known_hosts</code> value.</li>
<li>Use environments with required reviewers and deployment rules for production.</li>
<li>Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.</li>
<li>End every deploy with a real HTTP request against the public URL.</li>
<li>Rotate the deploy key on a schedule and whenever someone leaves.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is it safe to store an SSH private key in GitHub secrets?</h3>



<p class="wp-block-paragraph">Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.</p>



<h3 class="wp-block-heading">Should I use a ready-made SSH deploy action?</h3>



<p class="wp-block-paragraph">They work and they are convenient. My preference is plain <code>ssh</code> and <code>rsync</code> in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.</p>



<h3 class="wp-block-heading">Can I use OIDC instead of a stored key?</h3>



<p class="wp-block-paragraph">Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub&#8217;s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.</p>



<h3 class="wp-block-heading">Should I allowlist GitHub&#8217;s IP ranges on my firewall?</h3>



<p class="wp-block-paragraph">Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer&#8217;s runners share those ranges. Narrowing what the key can do achieves more.</p>



<h3 class="wp-block-heading">Self-hosted runner or hosted runner?</h3>



<p class="wp-block-paragraph">Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.</p>



<h3 class="wp-block-heading">How do I roll back a bad deploy?</h3>



<p class="wp-block-paragraph">Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.</p>



<h3 class="wp-block-heading">How often should the deploy key be rotated?</h3>



<p class="wp-block-paragraph">On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to <code>authorized_keys</code> and updating the secret, so there is no good reason for it to be a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.</p>



<p class="wp-block-paragraph">So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger&#8217;s code did with your credentials.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.</li>
<li>Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.</li>
<li>Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.</li>
<li>Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.</li>
<li>Pinning and auditing third-party actions, plus repository settings that limit what can run.</li>
<li>Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.</li>
</ul>



<p class="wp-block-paragraph">Send me your workflow file and the output of <code>ssh deploy@yourhost 'id'</code>, and I will tell you what a stolen key would get.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
