<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>OpenSSH | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/openssh/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/openssh/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 16:38:56 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>OpenSSH | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/openssh/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</title>
		<link>https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Ansible Collections]]></category>
		<category><![CDATA[Ansible Vault]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=204</guid>

					<description><![CDATA[<p>A single Ansible run can take a blank VPS to a hardened, reproducible server. The hard part is not the tasks, it is the order. Here is the ordering that keeps you logged in, the failures that pass green and bite two weeks later, and how to make the second run as safe as the first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.</p>



<p class="wp-block-paragraph">That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report <code>changed</code>, do nothing useful, and leave you believing the server is hardened when it is not.</p>



<p class="wp-block-paragraph">This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.</p>



<h2 class="wp-block-heading">What a one-run bootstrap actually has to survive</h2>



<p class="wp-block-paragraph">A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.</p>



<p class="wp-block-paragraph">Four things break it, and they break it in different ways:</p>



<ul class="wp-block-list">
<li><strong>The chicken-and-egg problem.</strong> Ansible needs Python on the target to run almost any module, and a minimal image may not have it.</li>

<li><strong>Lockout.</strong> You remove your own access before proving the replacement access works.</li>

<li><strong>Non-idempotent tasks.</strong> The first run is fine. The second run is where the damage happens.</li>

<li><strong>Missing collections.</strong> The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with <code>ansible-core</code>.</li>
</ul>



<p class="wp-block-paragraph">Everything below is organised around those four, because that is how they actually show up.</p>



<h2 class="wp-block-heading">The chicken-and-egg problem: no Python, no modules</h2>



<p class="wp-block-paragraph">Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.</p>



<p class="wp-block-paragraph">The escape hatch is <code>ansible.builtin.raw</code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with <code>gather_facts: false</code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.</p>



<pre class="wp-block-code"><code>- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update &amp;&amp; apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:</code></pre>



<p class="wp-block-paragraph">Two details worth understanding rather than copying. <code>failed_when: false</code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And <code>ansible.builtin.setup</code> is the fact-gathering module called explicitly, which means from that point on <code>ansible_distribution</code> and friends are available even though the play started with gathering disabled.</p>



<p class="wp-block-paragraph">If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.</p>



<h2 class="wp-block-heading">Failure family one: locking yourself out while the playbook says ok</h2>



<p class="wp-block-paragraph">This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.</p>



<h3 class="wp-block-heading">The order that survives</h3>



<p class="wp-block-paragraph">The rule is simple: <strong>nothing that removes access happens until something has proven the new access works.</strong> In practice:</p>



<ol class="wp-block-list">
<li>Connect as whatever the provider handed you, usually <code>root</code> or a sudo-capable default user.</li>

<li>Create the admin account, add it to the sudo or wheel group, install its authorized key.</li>

<li>Write the sudoers drop-in and validate it with <code>visudo -cf</code>.</li>

<li><strong>Prove you can log in as that account and escalate.</strong> This is the step everyone skips.</li>

<li>Install the firewall and allow SSH <em>before</em> enabling it.</li>

<li>Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.</li>

<li>Only now disable root login and password authentication.</li>
</ol>



<p class="wp-block-paragraph">Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:</p>



<pre class="wp-block-code"><code>- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"</code></pre>



<p class="wp-block-paragraph">Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.</p>



<h3 class="wp-block-heading">The socket activation trap</h3>



<p class="wp-block-paragraph">This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. <code>ssh.socket</code> owns the listening port, and <code>Port</code> in <code>sshd_config</code> is not what decides where the daemon listens.</p>



<p class="wp-block-paragraph">So your playbook writes <code>Port 2222</code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.</p>



<p class="wp-block-paragraph">The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:</p>



<pre class="wp-block-code"><code># Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'</code></pre>



<p class="wp-block-paragraph">If <code>ssh.socket</code> is enabled, the override needs an empty <code>ListenStream=</code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.</p>



<p class="wp-block-paragraph">Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.</p>



<h3 class="wp-block-heading">Firewall ordering</h3>



<p class="wp-block-paragraph">UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an <code>UNREACHABLE</code> that will not recover. Rules go in first, always:</p>



<pre class="wp-block-code"><code>- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled</code></pre>



<p class="wp-block-paragraph">Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.</p>



<h2 class="wp-block-heading">Failure family two: the second run is the dangerous one</h2>



<p class="wp-block-paragraph">A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are <code>shell</code> and <code>command</code> tasks with no guard.</p>



<p class="wp-block-paragraph">An unguarded <code>shell</code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.</p>



<p class="wp-block-paragraph">Guard them properly:</p>



<ul class="wp-block-list">
<li><code>creates:</code> or <code>removes:</code> makes the task skip when the target state already exists. Prefer this.</li>

<li><code>changed_when: false</code> for read-only commands so they stop lying about changing things.</li>

<li><code>when:</code> driven by a registered check, for anything conditional.</li>
</ul>



<p class="wp-block-paragraph">The other idempotency trap is <code>lineinfile</code> against <code>sshd_config</code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:</p>



<pre class="wp-block-code"><code>- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false</code></pre>



<p class="wp-block-paragraph">Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so <code>validate:</code> on the template task with <code>sshd -t -f %s</code> will fail on a temporary file that has no host keys and no context. Run <code>sshd -t</code> as its own task afterwards instead, and let the handler restart only if it passes.</p>



<p class="wp-block-paragraph">Two prerequisites for the drop-in approach. The base <code>sshd_config</code> needs an <code>Include /etc/ssh/sshd_config.d/*.conf</code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the <em>first</em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.</p>



<h2 class="wp-block-heading">Failure family three: the collections are not where you think</h2>



<p class="wp-block-paragraph"><code>ansible-core</code> ships the <code>ansible.builtin</code> namespace and essentially nothing else. The full <code>ansible</code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.</p>



<p class="wp-block-paragraph">Things people assume are built in and are not: <code>ufw</code> and <code>timezone</code> live in <code>community.general</code>. <code>authorized_key</code>, <code>sysctl</code> and <code>mount</code> live in <code>ansible.posix</code>. Declare them and pin them:</p>



<pre class="wp-block-code"><code># requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml</code></pre>



<p class="wp-block-paragraph">Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. <code>ansible-lint</code> flags this and it is worth listening to.</p>



<h2 class="wp-block-heading">Failure family four: secrets in the repository</h2>



<p class="wp-block-paragraph">Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.</p>



<p class="wp-block-paragraph">Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:</p>



<pre class="wp-block-code"><code>ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass</code></pre>



<p class="wp-block-paragraph">The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Putting it together: how to bootstrap a VPS with Ansible end to end</h2>



<p class="wp-block-paragraph">Structure matters less than order, but a shape that works well is a single <code>site.yml</code> calling roles in dependency order, with the risky access changes fenced off at the end.</p>



<pre class="wp-block-code"><code># inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22</code></pre>



<p class="wp-block-paragraph">Then the run itself, in this order:</p>



<ol class="wp-block-list">
<li><strong>Reachability.</strong> Python check, fact gathering, timezone, hostname, package cache refresh.</li>

<li><strong>Identity.</strong> Admin user, group membership, authorized key, sudoers drop-in with <code>visudo -cf</code> validation.</li>

<li><strong>Verification gate.</strong> The task shown earlier. Nothing destructive runs before this passes.</li>

<li><strong>Firewall.</strong> Rules first, policy second, enable third.</li>

<li><strong>Unattended updates.</strong> <code>unattended-upgrades</code> on Debian and Ubuntu, <code>dnf-automatic</code> with its systemd timer on RHEL-family hosts.</li>

<li><strong>Brute-force protection.</strong> fail2ban with your own <code>/etc/fail2ban/jail.local</code>, never edits to <code>jail.conf</code>.</li>

<li><strong>Workload.</strong> Docker, a web server, a monitoring agent, whatever the box is for.</li>

<li><strong>Access teardown.</strong> Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.</li>
</ol>



<p class="wp-block-paragraph">When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:</p>



<pre class="wp-block-code"><code>- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60</code></pre>



<p class="wp-block-paragraph"><code>meta: reset_connection</code> does not accept a <code>when</code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.</p>



<h2 class="wp-block-heading">Test it before you point it at anything real</h2>



<p class="wp-block-paragraph">The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.</p>



<pre class="wp-block-code"><code># catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff</code></pre>



<p class="wp-block-paragraph">One caveat on <code>--check</code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong><code>UNREACHABLE</code> on the very first task.</strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with <code>ssh-keygen -R</code> and re-add the correct one with <code>ssh-keyscan</code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.</li>

<li><strong><code>UNREACHABLE</code> immediately after a firewall task.</strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.</li>

<li><strong>&#8220;Missing sudo password&#8221;.</strong> The sudoers drop-in is not in effect or has bad syntax. Validate with <code>visudo -cf</code> on the file itself before trusting it.</li>

<li><strong>Module not found under an FQCN.</strong> The collection is not installed on the machine running the playbook. Confirm with <code>ansible-galaxy collection list</code>.</li>

<li><strong>New group membership ignored.</strong> Adding a user to <code>docker</code> mid-play does not affect the already-open connection. Insert <code>meta: reset_connection</code> after the group change.</li>

<li><strong>fail2ban installed but not banning.</strong> On images without rsyslog, <code>/var/log/auth.log</code> may not exist, so a jail configured to read it never starts. Check <code>fail2ban-client status</code> and switch the jail to the systemd backend if the log file is absent.</li>

<li><strong>Everything green, port unchanged.</strong> Socket activation. Check <code>systemctl is-enabled ssh.socket</code> and <code>ss -tlnp</code>.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling root login in the same play that creates the replacement user, with no verification step between them.</li>

<li>Enabling UFW before the SSH allow rule exists.</li>

<li>Assuming <code>Port</code> in <code>sshd_config</code> is authoritative on every distribution.</li>

<li>Unguarded <code>shell</code> tasks that notify handlers, causing service restarts on every run.</li>

<li>Editing <code>jail.conf</code> instead of creating <code>jail.local</code>, so the next package update silently reverts it.</li>

<li>Committing plaintext secrets because &#8220;it is only a bootstrap playbook&#8221;.</li>

<li>Testing on a host you cannot afford to lose, with no console access configured.</li>

<li>Relying on <code>--check</code> as a substitute for a real run on a disposable host.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.</li>

<li>Put a verification task before every irreversible one, and let the play fail there rather than after.</li>

<li>Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.</li>

<li>Validate configuration before restarting the service that reads it. <code>sshd -t</code>, <code>visudo -cf</code>, <code>nginx -t</code>.</li>

<li>Pin collections in <code>requirements.yml</code> and install them in CI, not just locally.</li>

<li>Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.</li>

<li>Reboot once during testing. Half of the silent failures only appear after a restart.</li>

<li>Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you really bootstrap a VPS with Ansible in a single playbook run?</h3>



<p class="wp-block-paragraph">Yes, including changing the SSH user and port mid-run. The mechanism is <code>set_fact</code> to update the connection variables, <code>meta: reset_connection</code> to drop the cached connection, and <code>wait_for_connection</code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.</p>



<h3 class="wp-block-heading">Should I use cloud-init or Ansible for initial server setup?</h3>



<p class="wp-block-paragraph">Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.</p>



<h3 class="wp-block-heading">Do I need Terraform or OpenTofu as well?</h3>



<p class="wp-block-paragraph">Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.</p>



<h3 class="wp-block-heading">Why does my playbook work the first time and fail the second?</h3>



<p class="wp-block-paragraph">Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.</p>



<h3 class="wp-block-heading">Is changing the SSH port worth doing?</h3>



<p class="wp-block-paragraph">It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.</p>



<h3 class="wp-block-heading">How do I recover if the playbook locks me out?</h3>



<p class="wp-block-paragraph">Through the provider&#8217;s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.</p>



<h3 class="wp-block-heading">Can this run against several servers at once?</h3>



<p class="wp-block-paragraph">Yes, but use <code>serial: 1</code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.</p>



<p class="wp-block-paragraph">Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a bootstrap playbook you can actually rerun?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:</p>



<ul class="wp-block-list">
<li>Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.</li>

<li>Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.</li>

<li>Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.</li>

<li>Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.</li>

<li>Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.</li>

<li>Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.</li>
</ul>



<p class="wp-block-paragraph">Send me the playbook, a recap from a second run, or the output of <code>ss -tlnp</code> from the host that is behaving oddly, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</title>
		<link>https://john-nessime.com/blog/system-administration/vps-hardening-checklist/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:24 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=62</guid>

					<description><![CDATA[<p>You set PasswordAuthentication no, restarted SSH, and moved on. Months later the logs show successful password logins, because a vendor drop-in file sorted earlier and won. A VPS setup and hardening checklist where every step comes with the command that proves it worked.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You spun up a new box, worked through a hardening guide, set <code>PasswordAuthentication no</code>, restarted SSH, and moved on. Six months later you are reading auth logs for an unrelated reason and there it is: successful password logins. Not attempts. Logins.</p>



<p class="wp-block-paragraph">The line you wrote was real. The file was saved. What you did not know is that your distro puts an <code>Include /etc/ssh/sshd_config.d/*.conf</code> near the top of <code>sshd_config</code>, that sshd uses the <em>first</em> value it finds for any given keyword, and that your cloud provider&#8217;s image dropped a <code>50-cloud-init.conf</code> in that directory setting <code>PasswordAuthentication yes</code>. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.</p>



<p class="wp-block-paragraph">You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.</p>



<p class="wp-block-paragraph">That is the thing most guides get wrong, and it is the idea this <strong>VPS hardening checklist</strong> is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.</p>



<p class="wp-block-paragraph">This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.</p>



<h2 class="wp-block-heading">Before you touch anything</h2>



<p class="wp-block-paragraph">Two minutes of preparation that saves an evening.</p>



<p class="wp-block-paragraph"><strong>Find your provider&#8217;s console before you need it.</strong> Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.</p>



<p class="wp-block-paragraph"><strong>Never close the session you are working in.</strong> Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.</p>



<p class="wp-block-paragraph">Then update the system, because everything else assumes current packages:</p>



<pre class="wp-block-code"><code>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo reboot   # if a kernel update landed</code></pre>



<h2 class="wp-block-heading">1. A real user, and keys instead of passwords</h2>



<p class="wp-block-paragraph">Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.</p>



<pre class="wp-block-code"><code># On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"</code></pre>



<pre class="wp-block-code"><code># On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10</code></pre>



<p class="wp-block-paragraph">Now open a second terminal and confirm <code>ssh deploy@203.0.113.10</code> works, and that <code>sudo -v</code> works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.</p>



<h2 class="wp-block-heading">2. SSH config, and why your edit might do nothing</h2>



<p class="wp-block-paragraph">This is the section that actually matters. Two mechanisms silently override what you write.</p>



<h3 class="wp-block-heading">Drop-in files and first-match-wins</h3>



<p class="wp-block-paragraph">Modern OpenSSH packages ship an <code>Include</code> line near the top of <code>/etc/ssh/sshd_config</code>, which pulls in every <code>.conf</code> file from <code>/etc/ssh/sshd_config.d/</code> in alphabetical order, before the rest of the main file is read. Combine that with sshd&#8217;s rule that the first value found for a keyword wins, and the consequence is specific: <strong>your drop-in has to sort earlier than the vendor&#8217;s, not later.</strong></p>



<p class="wp-block-paragraph">Most guides tell you to name it <code>99-something.conf</code>, which is the convention almost everywhere else in Linux and exactly backwards here.</p>



<pre class="wp-block-code"><code># /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy</code></pre>



<p class="wp-block-paragraph">A note on <code>KbdInteractiveAuthentication</code>: older guides and older OpenSSH call this <code>ChallengeResponseAuthentication</code>. Setting only <code>PasswordAuthentication no</code> can still leave a keyboard-interactive path open on some configurations, which is why both are here.</p>



<p class="wp-block-paragraph">Now the part nobody does. Validate the syntax, then read the effective config:</p>



<pre class="wp-block-code"><code># Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'</code></pre>



<p class="wp-block-paragraph">If <code>passwordauthentication yes</code> comes back, your file lost to something else in that directory. Look at what is in there with <code>ls /etc/ssh/sshd_config.d/</code> and either rename yours to sort earlier or edit the offending file directly.</p>



<h3 class="wp-block-heading">Socket activation, and the port that will not change</h3>



<p class="wp-block-paragraph">The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: <code>ssh.socket</code> holds the listening port and starts <code>sshd</code> only when a connection arrives. On those systems the <code>Port</code> and <code>ListenAddress</code> directives in <code>sshd_config</code> may be ignored entirely, because systemd owns the socket.</p>



<p class="wp-block-paragraph">People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:</p>



<pre class="wp-block-code"><code># -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp</code></pre>



<p class="wp-block-paragraph">If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under <code>/lib/systemd/system/</code>, which a package update will overwrite:</p>



<pre class="wp-block-code"><code>sudo systemctl edit ssh.socket</code></pre>



<pre class="wp-block-code"><code># In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222</code></pre>



<pre class="wp-block-code"><code>sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it</code></pre>



<p class="wp-block-paragraph">Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">3. Firewall, in the right order</h2>



<p class="wp-block-paragraph">The ordering here is the whole trick. Set the default policy and add your rules <em>before</em> you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.</p>



<pre class="wp-block-code"><code>sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose</code></pre>



<p class="wp-block-paragraph">Read that <code>status verbose</code> output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.</p>



<p class="wp-block-paragraph">One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with <code>-p 5432:5432</code> is reachable from the internet even though ufw says that port is denied. Bind container ports to <code>127.0.0.1</code> explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.</p>



<p class="wp-block-paragraph">A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.</p>



<h2 class="wp-block-heading">4. Automatic security updates</h2>



<p class="wp-block-paragraph">Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.</p>



<pre class="wp-block-code"><code>sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug</code></pre>



<p class="wp-block-paragraph">By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.</p>



<h2 class="wp-block-heading">5. fail2ban, and what it is actually for</h2>



<p class="wp-block-paragraph">Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.</p>



<p class="wp-block-paragraph">What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials <em>do</em> exist: a mail server, a web application login form, an FTP daemon.</p>



<pre class="wp-block-code"><code>sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd</code></pre>



<p class="wp-block-paragraph">Check that <code>status</code> output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, <code>journalctl -u fail2ban</code> will tell you why.</p>



<p class="wp-block-paragraph">Set <code>ignoreip</code> in <code>jail.local</code> to include your office or home address. Banning yourself is a rite of passage but it is avoidable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">6. The boring items that matter more than the exciting ones</h2>



<p class="wp-block-paragraph">Nobody writes blog posts about these, and they cause more real outages than any of the above.</p>



<h3 class="wp-block-heading">Provider account security</h3>



<p class="wp-block-paragraph">Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.</p>



<h3 class="wp-block-heading">Time and timezone</h3>



<pre class="wp-block-code"><code>sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"</code></pre>



<p class="wp-block-paragraph">UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.</p>



<h3 class="wp-block-heading">Swap on small instances</h3>



<pre class="wp-block-code"><code># A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h</code></pre>



<h3 class="wp-block-heading">Backups that leave the machine</h3>



<p class="wp-block-paragraph">Your provider&#8217;s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.</p>



<p class="wp-block-paragraph">Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man&#8217;s switch service at the backup job so you find out when it stops running, rather than the day you need it.</p>



<h3 class="wp-block-heading">Knowing the box is alive</h3>



<p class="wp-block-paragraph">At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.</p>



<h2 class="wp-block-heading">What I would skip</h2>



<p class="wp-block-paragraph">Not everything in the long hardening guides earns its cost on a single web server.</p>



<ul class="wp-block-list">
<li><strong>Large sysctl hardening blocks pasted from a gist.</strong> Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.</li>
<li><strong>AIDE, auditd and file-integrity monitoring</strong> on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.</li>
<li><strong>Rebuilding OpenSSH&#8217;s cipher list.</strong> Current defaults are good. Guides that hand you a <code>Ciphers</code> line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.</li>
<li><strong>Disabling ping.</strong> It breaks your own monitoring and hides nothing.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Locked out after an SSH change</h3>



<p class="wp-block-paragraph">Provider console, log in as root, and revert. If you cannot remember what you changed, <code>ls -lt /etc/ssh/sshd_config.d/</code> shows the most recently modified file. Run <code>sshd -t</code> to see whether the config is even valid, since a syntax error stops sshd starting at all.</p>



<h3 class="wp-block-heading">Key auth rejected, password prompt appears instead</h3>



<p class="wp-block-paragraph">Almost always permissions. The home directory must not be group-writable, <code>~/.ssh</code> needs 700 and <code>authorized_keys</code> 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run <code>ssh -v</code> to see how far it gets, and check <code>journalctl -u ssh</code> on the server for the actual reason.</p>



<h3 class="wp-block-heading">The setting I applied is not in effect</h3>



<p class="wp-block-paragraph"><code>sudo sshd -T</code> and search for the keyword. If the effective value differs from your file, something in <code>sshd_config.d/</code> sorts earlier, or a <code>Match</code> block later in the config is overriding it for your connection.</p>



<h3 class="wp-block-heading">A service is reachable that the firewall says is blocked</h3>



<p class="wp-block-paragraph">Docker, or another daemon writing netfilter rules directly. Check with <code>sudo iptables -S</code> or <code>sudo nft list ruleset</code>, and always verify from a machine outside the server rather than trusting <code>ufw status</code>.</p>



<h3 class="wp-block-heading">Nothing obviously wrong, but who has been logging in?</h3>



<pre class="wp-block-code"><code>last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Closing your only SSH session before testing the new configuration from a second one.</li>
<li>Enabling the firewall before adding the SSH rule.</li>
<li>Naming your sshd drop-in <code>99-</code>, so a vendor file that sorts earlier wins.</li>
<li>Trusting the file you edited instead of running <code>sshd -T</code>.</li>
<li>Generating the SSH key on the server.</li>
<li>Changing the SSH port and counting it as a security control.</li>
<li>Disabling password auth for SSH while leaving a web control panel with a weak password exposed.</li>
<li>No two-factor authentication on the hosting account that can reset root.</li>
<li>Treating provider snapshots as backups.</li>
<li>Installing fail2ban and never checking that it started.</li>
<li>Publishing container ports without binding them to localhost, then assuming ufw covers it.</li>
<li>Pasting a sysctl block you cannot explain line by line.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Verify every control after applying it, with a command that reads the running state.</li>
<li>Keep a second session open and the provider console tested before making SSH changes.</li>
<li>Key-only authentication, root login disabled, and an explicit <code>AllowUsers</code> list.</li>
<li>Default-deny inbound firewall, with rules added before the firewall is enabled.</li>
<li>Automatic security updates on, with a deliberate decision about automatic reboots.</li>
<li>Two-factor authentication on the hosting account and the registrar.</li>
<li>Backups off the machine, on separate credentials, and restore-tested at least once.</li>
<li>UTC everywhere, with clock sync confirmed.</li>
<li>Monitoring that pages you on disk usage before it pages you on downtime.</li>
<li>Script the whole build, so the next server is identical and you are not repeating this from memory.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the single most important step?</h3>



<p class="wp-block-paragraph">Key-only SSH with root login disabled, verified with <code>sshd -T</code>. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.</p>



<h3 class="wp-block-heading">Should I change the SSH port?</h3>



<p class="wp-block-paragraph">For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.</p>



<h3 class="wp-block-heading">Do I need fail2ban if password authentication is off?</h3>



<p class="wp-block-paragraph">Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.</p>



<h3 class="wp-block-heading">Why did my sshd_config change get ignored?</h3>



<p class="wp-block-paragraph">Two likely causes. A file in <code>/etc/ssh/sshd_config.d/</code> that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is <code>Port</code> or <code>ListenAddress</code> on a socket-activated system, where systemd owns the listener. <code>sshd -T</code> answers the first case and <code>ss -tlnp</code> answers the second.</p>



<h3 class="wp-block-heading">Is ufw enough, or should I use nftables directly?</h3>



<p class="wp-block-paragraph">ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.</p>



<h3 class="wp-block-heading">How do I harden a server I did not build?</h3>



<p class="wp-block-paragraph">Audit before you change anything. <code>sshd -T</code>, <code>ufw status verbose</code>, <code>ss -tlnp</code> for unexpected listeners, <code>systemctl list-units --type=service --state=running</code>, and the contents of every <code>authorized_keys</code> on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.</p>



<h3 class="wp-block-heading">Should this be automated?</h3>



<p class="wp-block-paragraph">Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Editing a config file is not the same as changing a system&#8217;s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.</p>



<p class="wp-block-paragraph">So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. <code>sshd -T</code> for SSH, <code>ufw status verbose</code> for the firewall, <code>ss -tlnp</code> for what is listening, <code>systemctl status</code> for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.</p>



<h2 class="wp-block-heading">Want a second pair of eyes on your server?</h2>



<p class="wp-block-paragraph">Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:</p>



<ul class="wp-block-list">
<li>Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.</li>
<li>Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.</li>
<li>Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.</li>
<li>Moving SSH off the public internet onto a private overlay network or bastion.</li>
<li>Backup strategy that survives the hosting account being compromised, including a tested restore.</li>
<li>Firewall and container networking, especially where published Docker ports are bypassing the firewall.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>sshd -T</code>, <code>ufw status verbose</code> and <code>ss -tlnp</code>, and I will tell you what stands out.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</title>
		<link>https://john-nessime.com/blog/devops/github-actions-vps-deployment/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:21 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=66</guid>

					<description><![CDATA[<p>A deploy key in GitHub secrets is a shell on your production server, handed to a container you don't control, running code from maintainers you've never met. Here's how to build the pipeline so a leaked key isn't worth much: forced commands, scoped sudo, pinned actions and a real approval gate.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Stop and look at what you are about to build. A private SSH key that can log into your production server, pasted into a text box on github.com, handed to a container you do not control, which then runs code from repositories maintained by people you have never met.</p>



<p class="wp-block-paragraph">That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer&#8217;s token, pushed a malicious commit, and then <em>retroactively repointed every version tag</em> at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner&#8217;s secrets into the build log. On public repositories, those logs are readable by anyone.</p>



<p class="wp-block-paragraph">Nobody&#8217;s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.</p>



<p class="wp-block-paragraph">A <strong>GitHub Actions VPS deployment</strong> is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.</p>



<h2 class="wp-block-heading">Who can actually reach that key</h2>



<p class="wp-block-paragraph">Worth being explicit, because it is a longer list than people expect:</p>



<ul class="wp-block-list">
<li><strong>Anyone with write access to the repository.</strong> A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.</li>
<li><strong>Every action in the job.</strong> Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.</li>
<li><strong>Anyone who can influence what runs.</strong> This is why <code>pull_request_target</code> combined with checking out the pull request&#8217;s own code is such a well-known foot-gun: it runs the fork&#8217;s code with the base repository&#8217;s secrets in scope.</li>
</ul>



<p class="wp-block-paragraph">The goal is not to make the key unreachable. It is to make a stolen key not worth much.</p>



<h2 class="wp-block-heading">Decide how the runner authenticates</h2>



<p class="wp-block-paragraph">Three options, and the right one depends on what you already run.</p>



<ol class="wp-block-list">
<li><strong>A dedicated SSH key in repository secrets.</strong> The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.</li>
<li><strong>An overlay network.</strong> Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.</li>
<li><strong>A self-hosted runner on the server.</strong> No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.</li>
</ol>



<p class="wp-block-paragraph">One thing that sounds appealing and is not practical: allowlisting GitHub&#8217;s runner IP ranges at your firewall. The ranges are published through GitHub&#8217;s meta API, but they are enormous and they change, and every other GitHub customer&#8217;s job runs from inside them. It buys you very little for a lot of maintenance.</p>



<h2 class="wp-block-heading">Narrow the key until it is boring</h2>



<p class="wp-block-paragraph">Assume the key leaks. What can the holder do? Work backwards from there.</p>



<p class="wp-block-paragraph"><strong>Give it its own user.</strong> A <code>deploy</code> account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.</p>



<p class="wp-block-paragraph"><strong>Restrict the key in <code>authorized_keys</code>.</strong> SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:</p>



<pre class="wp-block-code"><code># ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions</code></pre>



<p class="wp-block-paragraph">The guard script decides what the key is allowed to do. Keep it short enough to read in one go:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" &gt;&amp;2
    exit 1
    ;;
esac</code></pre>



<p class="wp-block-paragraph">Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.</p>



<p class="wp-block-paragraph"><strong>Scope the sudo entry too.</strong> The deploy user needs to reload a service, not run arbitrary commands as root:</p>



<pre class="wp-block-code"><code># visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx</code></pre>



<p class="wp-block-paragraph">Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted <code>deploy</code> user with no sudo is a reasonable stopping point.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Verify the server, not just the key</h2>



<p class="wp-block-paragraph">Almost every tutorial on this topic contains <code>StrictHostKeyChecking=no</code>, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.</p>



<pre class="wp-block-code"><code># Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com</code></pre>



<p class="wp-block-paragraph">Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.</p>



<h2 class="wp-block-heading">The workflow</h2>



<pre class="wp-block-code"><code>name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" &gt; ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" &gt; ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz &gt; /dev/null</code></pre>



<p class="wp-block-paragraph">Three details in there worth calling out, because they are the ones people leave out.</p>



<p class="wp-block-paragraph"><code>-o BatchMode=yes</code> stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.</p>



<p class="wp-block-paragraph"><code>--delete</code> is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if <code>.deployignore</code> is wrong. Run it once with <code>--dry-run</code> against staging and read the list before you point it at production.</p>



<p class="wp-block-paragraph">The final <code>curl --fail</code> is the difference between &#8220;the files copied&#8221; and &#8220;the site works&#8221;. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.</p>



<h2 class="wp-block-heading">Third-party actions</h2>



<p class="wp-block-paragraph">There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain <code>ssh</code> and <code>rsync</code> for that reason.</p>



<p class="wp-block-paragraph">Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.</p>



<p class="wp-block-paragraph">Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.</p>



<h2 class="wp-block-heading">Gate production properly</h2>



<p class="wp-block-paragraph">The <code>environment:</code> key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:</p>



<ul class="wp-block-list">
<li><strong>Scoped secrets.</strong> The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.</li>
<li><strong>Required reviewers.</strong> The job pauses and waits for a named human to approve it. This is the single control that turns &#8220;anyone with write access can deploy&#8221; into &#8220;anyone with write access can request a deploy&#8221;.</li>
<li><strong>Deployment branch and tag rules.</strong> Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.</li>
</ul>



<p class="wp-block-paragraph">Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.</p>



<p class="wp-block-paragraph">For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Permission denied (publickey)</h3>



<p class="wp-block-paragraph">The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses <code>printf '%sn'</code> rather than <code>echo</code>: it guarantees the newline regardless of what got stored.</p>



<p class="wp-block-paragraph">After that, check permissions on the server: <code>~/.ssh</code> must be 700 and <code>authorized_keys</code> 600, owned by the deploy user, and the home directory must not be group-writable. Add <code>-vvv</code> to the ssh command to see how far the handshake gets, and read <code>journalctl -u ssh</code> on the server for the reason, since the client is deliberately vague.</p>



<h3 class="wp-block-heading">Host key verification failed</h3>



<p class="wp-block-paragraph">Either the <code>known_hosts</code> variable is empty, or the server&#8217;s host key genuinely changed. If you rebuilt the server, regenerate it with <code>ssh-keyscan</code> and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.</p>



<h3 class="wp-block-heading">The job hangs until the timeout</h3>



<p class="wp-block-paragraph">SSH is waiting on a prompt nobody will answer. Add <code>-o BatchMode=yes</code> and it fails immediately with a readable reason instead.</p>



<h3 class="wp-block-heading">rsync works from my laptop, fails from the runner</h3>



<p class="wp-block-paragraph">If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records <code>$SSH_ORIGINAL_COMMAND</code>, run the job once, and read what rsync actually sent.</p>



<h3 class="wp-block-heading">Works on push, fails on pull requests</h3>



<p class="wp-block-paragraph">Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for <code>pull_request_target</code> to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.</p>



<h3 class="wp-block-heading">A secret appeared in the logs</h3>



<p class="wp-block-paragraph">Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using an existing personal SSH key instead of generating a dedicated one for the runner.</li>
<li>Giving the deploy user full sudo, or a sudo rule with a wildcard in it.</li>
<li>Disabling host key checking with <code>StrictHostKeyChecking=no</code>.</li>
<li>Referencing third-party actions by tag in the job that holds your production key.</li>
<li>Interpolating <code>${{ secrets.X }}</code> directly into a <code>run:</code> script instead of passing it through <code>env:</code>.</li>
<li>Leaving default write permissions on a job that only needs to read.</li>
<li>Using <code>cancel-in-progress: true</code> on a deploy, so a second push interrupts the first mid-copy.</li>
<li>Running <code>rsync --delete</code> at production without testing the exclude list.</li>
<li>Treating &#8220;the files copied&#8221; as a successful deploy, with no health check.</li>
<li>Deploying straight from branch pushes with no approval step.</li>
<li>Relying on log masking to keep a leaked secret out of the output.</li>
<li>Never rotating the deploy key, including after someone leaves the team.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Dedicated deploy user, dedicated key, used for nothing else.</li>
<li>Restrict the key in <code>authorized_keys</code> with <code>restrict</code> and a forced command.</li>
<li>Scope sudo to the exact commands the deploy needs.</li>
<li>Pin every action to a full commit SHA and let Dependabot bump them.</li>
<li>Keep the deploy job small: fewer steps means fewer things that can read the secret.</li>
<li>Set <code>permissions:</code> explicitly, defaulting to read-only.</li>
<li>Verify host keys from a stored <code>known_hosts</code> value.</li>
<li>Use environments with required reviewers and deployment rules for production.</li>
<li>Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.</li>
<li>End every deploy with a real HTTP request against the public URL.</li>
<li>Rotate the deploy key on a schedule and whenever someone leaves.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is it safe to store an SSH private key in GitHub secrets?</h3>



<p class="wp-block-paragraph">Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.</p>



<h3 class="wp-block-heading">Should I use a ready-made SSH deploy action?</h3>



<p class="wp-block-paragraph">They work and they are convenient. My preference is plain <code>ssh</code> and <code>rsync</code> in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.</p>



<h3 class="wp-block-heading">Can I use OIDC instead of a stored key?</h3>



<p class="wp-block-paragraph">Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub&#8217;s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.</p>



<h3 class="wp-block-heading">Should I allowlist GitHub&#8217;s IP ranges on my firewall?</h3>



<p class="wp-block-paragraph">Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer&#8217;s runners share those ranges. Narrowing what the key can do achieves more.</p>



<h3 class="wp-block-heading">Self-hosted runner or hosted runner?</h3>



<p class="wp-block-paragraph">Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.</p>



<h3 class="wp-block-heading">How do I roll back a bad deploy?</h3>



<p class="wp-block-paragraph">Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.</p>



<h3 class="wp-block-heading">How often should the deploy key be rotated?</h3>



<p class="wp-block-paragraph">On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to <code>authorized_keys</code> and updating the secret, so there is no good reason for it to be a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.</p>



<p class="wp-block-paragraph">So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger&#8217;s code did with your credentials.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.</li>
<li>Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.</li>
<li>Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.</li>
<li>Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.</li>
<li>Pinning and auditing third-party actions, plus repository settings that limit what can run.</li>
<li>Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.</li>
</ul>



<p class="wp-block-paragraph">Send me your workflow file and the output of <code>ssh deploy@yourhost 'id'</code>, and I will tell you what a stolen key would get.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
