The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.<\/p>\n\n\n\n
That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.<\/p>\n\n\n\n A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.<\/p>\n\n\n\n Four things break it, and they break it in different ways:<\/p>\n\n\n\n Everything below is organised around those four, because that is how they actually show up.<\/p>\n\n\n\n Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.<\/p>\n\n\n\n The escape hatch is Two details worth understanding rather than copying. If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.<\/p>\n\n\n\n This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.<\/p>\n\n\n\n The rule is simple: nothing that removes access happens until something has proven the new access works.<\/strong> In practice:<\/p>\n\n\n\n Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:<\/p>\n\n\n\n Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.<\/p>\n\n\n\n This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. So your playbook writes The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:<\/p>\n\n\n\n If Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.<\/p>\n\n\n\n UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.<\/p>\n\n\n\n A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are An unguarded Guard them properly:<\/p>\n\n\n\n The other idempotency trap is Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so Two prerequisites for the drop-in approach. The base Things people assume are built in and are not: Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.<\/p>\n\n\n\n Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:<\/p>\n\n\n\n The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.<\/p>\n\n\n\n Structure matters less than order, but a shape that works well is a single Then the run itself, in this order:<\/p>\n\n\n\n When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:<\/p>\n\n\n\n The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.<\/p>\n\n\n\n One caveat on Yes, including changing the SSH user and port mid-run. The mechanism is Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.<\/p>\n\n\n\n Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.<\/p>\n\n\n\n Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.<\/p>\n\n\n\n It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.<\/p>\n\n\n\n Through the provider’s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.<\/p>\n\n\n\n Yes, but use When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.<\/p>\n\n\n\n Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.<\/p>\n\n\n\n Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:<\/p>\n\n\n\n Send me the playbook, a recap from a second run, or the output of changed<\/code>, do nothing useful, and leave you believing the server is hardened when it is not.<\/p>\n\n\n\nWhat a one-run bootstrap actually has to survive<\/h2>\n\n\n\n
\n
ansible-core<\/code>.<\/li>\n<\/ul>\n\n\n\nThe chicken-and-egg problem: no Python, no modules<\/h2>\n\n\n\n
ansible.builtin.raw<\/code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with gather_facts: false<\/code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.<\/p>\n\n\n\n- name: Make the host manageable\n hosts: new_servers\n gather_facts: false\n become: true\n tasks:\n - name: Check whether a Python interpreter exists\n ansible.builtin.raw: test -e \/usr\/bin\/python3\n register: python_present\n changed_when: false\n failed_when: false\n\n - name: Install Python if it is missing\n ansible.builtin.raw: apt-get update && apt-get install -y python3\n when: python_present.rc != 0\n changed_when: true\n\n - name: Gather facts now that we can\n ansible.builtin.setup:<\/code><\/pre>\n\n\n\nfailed_when: false<\/code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And ansible.builtin.setup<\/code> is the fact-gathering module called explicitly, which means from that point on ansible_distribution<\/code> and friends are available even though the play started with gathering disabled.<\/p>\n\n\n\nFailure family one: locking yourself out while the playbook says ok<\/h2>\n\n\n\n
The order that survives<\/h3>\n\n\n\n
\n
root<\/code> or a sudo-capable default user.<\/li>\n\nvisudo -cf<\/code>.<\/li>\n\n- name: Prove the new admin account can log in and escalate\n ansible.builtin.command: id -u\n become: true\n changed_when: false\n vars:\n ansible_user: \"{{ admin_user }}\"\n ansible_ssh_private_key_file: \"{{ admin_key_path }}\"<\/code><\/pre>\n\n\n\nThe socket activation trap<\/h3>\n\n\n\n
ssh.socket<\/code> owns the listening port, and Port<\/code> in sshd_config<\/code> is not what decides where the daemon listens.<\/p>\n\n\n\nPort 2222<\/code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.<\/p>\n\n\n\n# Is the socket unit in play on this host?\nsystemctl is-enabled ssh.socket\n\n# What is actually listening, and under which unit?\nss -tlnp | grep -E 'ssh|:22'<\/code><\/pre>\n\n\n\nssh.socket<\/code> is enabled, the override needs an empty ListenStream=<\/code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.<\/p>\n\n\n\nFirewall ordering<\/h3>\n\n\n\n
UNREACHABLE<\/code> that will not recover. Rules go in first, always:<\/p>\n\n\n\n- name: Allow SSH before the firewall is armed\n community.general.ufw:\n rule: allow\n port: \"{{ ssh_port }}\"\n proto: tcp\n\n- name: Set default inbound policy\n community.general.ufw:\n direction: incoming\n policy: deny\n\n- name: Enable UFW\n community.general.ufw:\n state: enabled<\/code><\/pre>\n\n\n\nFailure family two: the second run is the dangerous one<\/h2>\n\n\n\n
shell<\/code> and command<\/code> tasks with no guard.<\/p>\n\n\n\nshell<\/code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.<\/p>\n\n\n\n\n
creates:<\/code> or removes:<\/code> makes the task skip when the target state already exists. Prefer this.<\/li>\n\nchanged_when: false<\/code> for read-only commands so they stop lying about changing things.<\/li>\n\nwhen:<\/code> driven by a registered check, for anything conditional.<\/li>\n<\/ul>\n\n\n\nlineinfile<\/code> against sshd_config<\/code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:<\/p>\n\n\n\n- name: Write the sshd hardening drop-in\n ansible.builtin.template:\n src: 99-hardening.conf.j2\n dest: \/etc\/ssh\/sshd_config.d\/99-hardening.conf\n owner: root\n group: root\n mode: \"0600\"\n notify: Restart sshd\n\n- name: Validate the full sshd configuration\n ansible.builtin.command: sshd -t\n changed_when: false<\/code><\/pre>\n\n\n\nvalidate:<\/code> on the template task with sshd -t -f %s<\/code> will fail on a temporary file that has no host keys and no context. Run sshd -t<\/code> as its own task afterwards instead, and let the handler restart only if it passes.<\/p>\n\n\n\nsshd_config<\/code> needs an Include \/etc\/ssh\/sshd_config.d\/*.conf<\/code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the first<\/em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.<\/p>\n\n\n\nFailure family three: the collections are not where you think<\/h2>\n\n\n\n
ansible-core<\/code> ships the ansible.builtin<\/code> namespace and essentially nothing else. The full ansible<\/code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.<\/p>\n\n\n\nufw<\/code> and timezone<\/code> live in community.general<\/code>. authorized_key<\/code>, sysctl<\/code> and mount<\/code> live in ansible.posix<\/code>. Declare them and pin them:<\/p>\n\n\n\n# requirements.yml\ncollections:\n - name: community.general\n - name: ansible.posix\n\n# install before the run, and in CI\nansible-galaxy collection install -r requirements.yml<\/code><\/pre>\n\n\n\nansible-lint<\/code> flags this and it is worth listening to.<\/p>\n\n\n\nFailure family four: secrets in the repository<\/h2>\n\n\n\n
ansible-vault encrypt group_vars\/all\/vault.yml\nansible-playbook site.yml --ask-vault-pass<\/code><\/pre>\n\n\n\n
\n\n\n\nPutting it together: how to bootstrap a VPS with Ansible end to end<\/h2>\n\n\n\n
site.yml<\/code> calling roles in dependency order, with the risky access changes fenced off at the end.<\/p>\n\n\n\n# inventory.yml\nnew_servers:\n hosts:\n web01:\n ansible_host: 203.0.113.10\n ansible_user: root\n ansible_port: 22\n vars:\n admin_user: deploy\n ssh_port: 22<\/code><\/pre>\n\n\n\n\n
visudo -cf<\/code> validation.<\/li>\n\nunattended-upgrades<\/code> on Debian and Ubuntu, dnf-automatic<\/code> with its systemd timer on RHEL-family hosts.<\/li>\n\n\/etc\/fail2ban\/jail.local<\/code>, never edits to jail.conf<\/code>.<\/li>\n\n- name: Point subsequent tasks at the new identity\n ansible.builtin.set_fact:\n ansible_user: \"{{ admin_user }}\"\n ansible_port: \"{{ ssh_port }}\"\n\n- name: Drop the cached connection\n ansible.builtin.meta: reset_connection\n\n- name: Confirm the host answers on the new settings\n ansible.builtin.wait_for_connection:\n timeout: 60<\/code><\/pre>\n\n\n\nmeta: reset_connection<\/code> does not accept a when<\/code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.<\/p>\n\n\n\nTest it before you point it at anything real<\/h2>\n\n\n\n
# catch YAML and structural errors instantly\nansible-playbook site.yml --syntax-check\n\n# style and correctness, including FQCN and idempotency smells\nansible-lint\n\n# show what would change without changing it\nansible-playbook -i inventory.yml site.yml --check --diff<\/code><\/pre>\n\n\n\n--check<\/code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.<\/p>\n\n\n\nTroubleshooting<\/h2>\n\n\n\n
\n
UNREACHABLE<\/code> on the very first task.<\/strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with ssh-keygen -R<\/code> and re-add the correct one with ssh-keyscan<\/code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.<\/li>\n\nUNREACHABLE<\/code> immediately after a firewall task.<\/strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.<\/li>\n\nvisudo -cf<\/code> on the file itself before trusting it.<\/li>\n\nansible-galaxy collection list<\/code>.<\/li>\n\ndocker<\/code> mid-play does not affect the already-open connection. Insert meta: reset_connection<\/code> after the group change.<\/li>\n\n\/var\/log\/auth.log<\/code> may not exist, so a jail configured to read it never starts. Check fail2ban-client status<\/code> and switch the jail to the systemd backend if the log file is absent.<\/li>\n\nsystemctl is-enabled ssh.socket<\/code> and ss -tlnp<\/code>.<\/li>\n<\/ul>\n\n\n\nCommon mistakes<\/h2>\n\n\n\n
\n
Port<\/code> in sshd_config<\/code> is authoritative on every distribution.<\/li>\n\nshell<\/code> tasks that notify handlers, causing service restarts on every run.<\/li>\n\njail.conf<\/code> instead of creating jail.local<\/code>, so the next package update silently reverts it.<\/li>\n\n--check<\/code> as a substitute for a real run on a disposable host.<\/li>\n<\/ul>\n\n\n\nBest practices<\/h2>\n\n\n\n
\n
sshd -t<\/code>, visudo -cf<\/code>, nginx -t<\/code>.<\/li>\n\nrequirements.yml<\/code> and install them in CI, not just locally.<\/li>\n\nFrequently asked questions<\/h2>\n\n\n\n
Can you really bootstrap a VPS with Ansible in a single playbook run?<\/h3>\n\n\n\n
set_fact<\/code> to update the connection variables, meta: reset_connection<\/code> to drop the cached connection, and wait_for_connection<\/code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.<\/p>\n\n\n\nShould I use cloud-init or Ansible for initial server setup?<\/h3>\n\n\n\n
Do I need Terraform or OpenTofu as well?<\/h3>\n\n\n\n
Why does my playbook work the first time and fail the second?<\/h3>\n\n\n\n
Is changing the SSH port worth doing?<\/h3>\n\n\n\n
How do I recover if the playbook locks me out?<\/h3>\n\n\n\n
Can this run against several servers at once?<\/h3>\n\n\n\n
serial: 1<\/code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.<\/p>\n\n\n\nThe one thing worth remembering<\/h2>\n\n\n\n
\n\n\n\nNeed a bootstrap playbook you can actually rerun?<\/h2>\n\n\n\n
\n
ss -tlnp<\/code> from the host that is behaving oddly, and I will tell you what I would change first.<\/p>\n\n\n\n