<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Server Provisioning | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/server-provisioning/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/server-provisioning/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 12:48:41 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Server Provisioning | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/server-provisioning/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</title>
		<link>https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Decommissioning]]></category>
		<category><![CDATA[DirectAdmin]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[search-replace]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SPF]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Website Migration]]></category>
		<category><![CDATA[WordPress Hosting]]></category>
		<category><![CDATA[WordPress Migration]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=301</guid>

					<description><![CDATA[<p>Moving one WordPress site is easy. Moving twenty in a weekend is a pipeline, and the failures that cost you are the quiet ones: writes landing on the old server, cron firing on two boxes, certificates blocked by ACME rate limits. Here is the fleet migration process I use, from manifest to per-site rollback.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message arrived on Monday morning. Three quote requests submitted over the weekend, confirmed by the people who submitted them, and none of them in the CRM. The site was up. The contact form worked when I tested it. Nothing in the error log.</p>



<p class="wp-block-paragraph">Those submissions went to the old server. It was still answering for a slice of the internet twenty hours after the DNS change, and by then I had already stopped syncing it.</p>



<p class="wp-block-paragraph">That is the failure that separates a bulk WordPress migration from moving a single site. When you move one site you watch it. When you move twenty in a weekend, you are running a pipeline, and the things that go wrong are the ones nobody is watching: writes landing on a server you have mentally decommissioned, cron firing on two boxes at once, a certificate that never issued because a DNS record was still pointing at the old IP when the challenge ran.</p>



<p class="wp-block-paragraph">This is the process I use for fleet moves onto a single VPS: the manifest that makes the weekend repeatable, the two-pass sync, the database step people rush, the certificate limits that bite specifically at fleet scale, and a cutover order that keeps rollback cheap. It assumes SSH on both boxes and some comfort on the command line.</p>



<h2 class="wp-block-heading">The failure that actually costs you: writes landing on the old server</h2>



<p class="wp-block-paragraph">Every migration guide tells you to lower your TTL. Almost none of them tell you what happens in the gap afterwards, and the gap is where the money goes.</p>



<p class="wp-block-paragraph">Here is the mechanism. You change the A record. Most resolvers pick up the new IP inside the TTL window. Some do not: corporate resolvers that clamp minimum TTLs, ISP resolvers that ignore short values, browsers holding a connection open, a mail server that resolved the host an hour ago. For those clients, the old server is still the site. It still has a working database. It still accepts comments, form entries, orders, user registrations and password resets.</p>



<p class="wp-block-paragraph">Nothing errors. Nothing logs. The write succeeds on a database you are about to throw away.</p>



<p class="wp-block-paragraph">There are only two honest answers to this, and picking one per site is the first real decision of the weekend.</p>



<ul class="wp-block-list">
<li><strong>Accept the loss.</strong> Fine for brochure sites, documentation, most blogs. A missed comment is not a business problem.</li>
<li><strong>Make the old server read-only the moment DNS changes.</strong> Mandatory for anything with a checkout, a booking flow, a members area or a lead form that feeds a sales process.</li>
</ul>



<p class="wp-block-paragraph">Read-only in practice means maintenance mode on the old copy, a static holding page on the old vhost, or pointing the old install&#8217;s <code>wp-config.php</code> at a database user with SELECT only. Any of those turns silent data loss into a visible error the visitor can react to. Someone who sees &#8220;temporarily unavailable&#8221; comes back. Someone whose order vanished does not.</p>



<p class="wp-block-paragraph">Do not skip this because the old server &#8220;will be off soon&#8221;. The window between DNS change and old server shutdown is the longest, least supervised part of the whole weekend.</p>



<h2 class="wp-block-heading">Build the fleet manifest before you touch a server</h2>



<p class="wp-block-paragraph">Twenty sites is the point where memory stops working. You will not remember which one runs an old PHP branch, which one has a hardcoded absolute path in a theme, which one has MX records pointing somewhere else entirely.</p>



<p class="wp-block-paragraph">So the first artifact is a table. One row per site, filled in before anything moves.</p>



<ul class="wp-block-list">
<li>Document root and database name</li>
<li>Current <code>siteurl</code> and <code>home</code> values, exactly as stored</li>
<li>PHP version the site currently runs under</li>
<li>Disk size of the document root and dump size of the database</li>
<li>Where DNS is authoritative, and who holds the login</li>
<li>Whether mail for that domain is on this server or elsewhere</li>
<li>Does it take writes from the public: comments, forms, checkout, registration</li>
<li>Cache and object cache in use</li>
<li>Cutover batch number and rollback owner</li>
</ul>



<p class="wp-block-paragraph">Most of that you can pull off the old server rather than typing it. This walks a DirectAdmin-style layout and prints a tab-separated line per WordPress install:</p>



<pre class="wp-block-code"><code>for d in /home/*/domains/*/public_html; do
  [ -f "$d/wp-config.php" ] || continue
  printf '%st%st%sn' 
    "$d" 
    "$(wp --path="$d" option get siteurl --skip-plugins --skip-themes 2&gt;/dev/null)" 
    "$(du -sh "$d" | cut -f1)"
done</code></pre>



<p class="wp-block-paragraph">The <code>--skip-plugins --skip-themes</code> flags matter here. Without them WP-CLI boots the full site, and one broken plugin on one install will stop your inventory halfway through with a fatal error.</p>



<p class="wp-block-paragraph">The DNS column is the one people fill in last and regret. Check where each zone actually lives before the weekend, not during it. A domain whose nameservers point at Cloudflare while you are editing records in the old host&#8217;s DNS panel will not move no matter how many times you refresh.</p>



<h2 class="wp-block-heading">Rebuild the stack, don&#8217;t clone the disk</h2>



<p class="wp-block-paragraph">There is a tempting shortcut where you image the old server and restore it onto the new one. It usually works and it is usually the wrong call, because you have just carried across every accumulated fix, every orphaned config, and every reason the old box got slow enough to need replacing.</p>



<p class="wp-block-paragraph">Build the destination clean. Whatever the provider is, whether that is Contabo, InterServer, Hetzner or a machine you already have, the sequence is the same: base OS, web server, PHP with the version and extensions the manifest says you need, database server, control panel if you use one, firewall, then a hardening pass.</p>



<p class="wp-block-paragraph">Two things to get right before any files land.</p>



<p class="wp-block-paragraph"><strong>PHP versions.</strong> If three sites need an older branch and seventeen are happy on current, install both and make the per-site assignment explicit in the manifest. Discovering the mismatch after cutover means a white screen on a live domain instead of a note in a spreadsheet. Check the extension list too, not just the version number, because <code>imagick</code>, <code>soap</code> and <code>intl</code> are the usual absentees.</p>



<p class="wp-block-paragraph"><strong>Access during the weekend.</strong> If you lock SSH and the control panel behind an IP allowlist, decide beforehand what your source address will be. A rotating home IP is how people lock themselves out of their own migration at two in the morning. A dedicated-IP option from a VPN provider such as NordVPN or Surfshark gives you one stable address to allowlist; a second admin path on a non-standard port works too. Deciding at the moment you are locked out does not.</p>



<h2 class="wp-block-heading">Two-pass rsync beats one big copy</h2>



<p class="wp-block-paragraph">The mistake at fleet scale is treating the file copy as a cutover-day task. Copy everything on cutover day and your maintenance window is however long twenty document roots take, which for a media-heavy fleet is hours you do not have.</p>



<p class="wp-block-paragraph">Split it. Do a warm sync days in advance while both sites are live, then a delta sync at cutover that moves only what changed.</p>



<p class="wp-block-paragraph">Warm sync, run from the old server, no deletes:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph">What those flags buy you: <code>-a</code> preserves permissions, times, symlinks and ownership; <code>-H</code> keeps hard links, which some backup plugins and dedup schemes rely on; <code>-A</code> and <code>-X</code> carry ACLs and extended attributes; <code>--numeric-ids</code> stops rsync remapping ownership through name lookups when the UIDs differ between boxes, which is exactly what happens when the new server creates users in a different order.</p>



<p class="wp-block-paragraph">Delta sync at cutover, same command with one addition:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --delete --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph"><code>--delete</code> removes files on the destination that no longer exist on the source. It is what makes the second pass an accurate mirror rather than a merge of two states. It is also the flag that will destroy your afternoon if you get the paths wrong, so check the trailing slashes: a source path ending in a slash copies the contents of the directory, one without it copies the directory itself, one level deeper than you meant.</p>



<p class="wp-block-paragraph">Run the delta pass once with <code>--dry-run</code> before running it for real. And on SELinux systems, relabel the tree afterwards: files arriving over rsync carry contexts the web server may not accept, and that produces a 403 which has nothing to do with permission bits.</p>



<h2 class="wp-block-heading">Databases last, and never with sed</h2>



<p class="wp-block-paragraph">Files can be stale for a few days. Databases cannot. Dump each database as late as you reasonably can, immediately before that site&#8217;s cutover, not on Friday for a Sunday move.</p>



<pre class="wp-block-code"><code># on the old server, per site
wp --path=/home/user/domains/example.com/public_html db export 
  /root/migration/example.com.sql --skip-plugins --skip-themes</code></pre>



<p class="wp-block-paragraph">Import on the new box, then handle URLs. If the domain is not changing, you may have nothing to replace. If the site is moving from HTTP to HTTPS, or from a staging hostname, or from a temporary preview domain you used for testing, you do.</p>



<p class="wp-block-paragraph">The rule that matters: do not run a find-and-replace over the SQL file with <code>sed</code>. WordPress stores widget settings, theme options, page builder layouts and plugin config as PHP serialized arrays, and every serialized string carries a byte-length prefix. Change the string without recalculating the prefix and the array fails to unserialize. The site does not error loudly. It loses its widgets, or its theme options revert to defaults, or the page builder renders an empty page.</p>



<p class="wp-block-paragraph">WP-CLI unserializes, replaces and reserializes, which is why it is the only tool I use for this.</p>



<pre class="wp-block-code"><code>wp search-replace 'http://example.com' 'https://example.com' 
  --all-tables-with-prefix 
  --precise 
  --skip-columns=guid 
  --report-changed-only 
  --dry-run</code></pre>



<p class="wp-block-paragraph">Read the dry-run output before you commit. Each flag earns its place: <code>--all-tables-with-prefix</code> catches plugin tables that share your prefix and would otherwise be skipped, <code>--precise</code> forces a full unserialize pass on every column instead of the faster string-level path, and <code>--skip-columns=guid</code> leaves the GUID column alone because feed readers use it as an identifier, not as a URL to fetch. Drop <code>--dry-run</code> to apply.</p>



<p class="wp-block-paragraph">Flush the object cache afterwards if the site uses one. A stale Redis or Memcached entry will happily serve you the old values and convince you the replacement failed.</p>



<h2 class="wp-block-heading">TLS at fleet scale: where Let&#8217;s Encrypt rate limits actually bite</h2>



<p class="wp-block-paragraph">Twenty certificates in one weekend is where a limit you have never noticed becomes the thing that blocks you.</p>



<p class="wp-block-paragraph">Let&#8217;s Encrypt publishes its limits and they are worth reading properly before a fleet move. Two of them are the ones that catch migrations.</p>



<ul class="wp-block-list">
<li><strong>New certificates per registered domain.</strong> Up to 50 certificates per registered domain every 7 days, counted globally across all accounts. If your twenty sites are twenty separate registered domains, you are nowhere near it. If they are subdomains of one or two parent domains, which is common for an agency fleet, you can burn through it.</li>
<li><strong>Authorization failures per identifier per account.</strong> Up to 5 failures per hostname per account per hour, refilling at one every 12 minutes. This is the one that actually bites, because it is triggered by the exact thing you are doing: requesting a certificate for a hostname whose DNS still points at the old server, so the HTTP-01 challenge is served by the wrong machine and fails.</li>
</ul>



<p class="wp-block-paragraph">Order matters as a result. If you issue before DNS moves, HTTP-01 fails. If you issue after DNS moves, visitors hit a certificate error in the gap. Three ways out, in the order I reach for them:</p>



<ol class="wp-block-list">
<li><strong>DNS-01 validation.</strong> Proves control through a TXT record instead of an HTTP request, so it does not care where the A record points. You can issue every certificate before touching DNS. This is the clean answer if your DNS provider has an API your ACME client supports.</li>
<li><strong>Cut DNS first, issue immediately after.</strong> Simple, and the exposure is short if you are watching. Batch it so you are never waiting on twenty challenges at once.</li>
<li><strong>Copy the existing certificates and keys across.</strong> Buys you a bridge until the new server can renew on its own. Only viable if you control the old server&#8217;s filesystem and the certificates have real time left.</li>
</ol>



<p class="wp-block-paragraph">Whichever you choose, test the plumbing against the staging environment first. Most ACME clients expose this as a dry-run mode, and it exists precisely so you can fail as many times as you like without spending production budget.</p>



<p class="wp-block-paragraph">One more thing that catches people: if any site has HSTS enabled with a long max-age, browsers that have seen it will refuse to fall back to HTTP. There is no grace period and no way to talk them out of it. Those sites need a valid certificate on the new server from the first request, which pushes them firmly toward DNS-01 or a copied certificate.</p>



<h2 class="wp-block-heading">Planning the cutover order for a bulk WordPress migration</h2>



<p class="wp-block-paragraph">Do not move twenty sites at once. Batch them, and batch by risk rather than alphabetically or by size.</p>



<ol class="wp-block-list">
<li><strong>Batch one: the throwaways.</strong> Two or three low-traffic sites you own or that nobody would notice. This batch exists to prove the stack, not to make progress.</li>
<li><strong>Batch two: the ordinary majority.</strong> Standard brochure and content sites, no checkout, no logins that matter.</li>
<li><strong>Batch three: the ones with writes.</strong> Stores, membership sites, anything with a form feeding a sales process.</li>
<li><strong>Batch four: the awkward ones.</strong> Custom stacks, odd PHP requirements, third-party integrations that whitelist your server IP.</li>
</ol>



<p class="wp-block-paragraph">Stop between batches. Look at the error logs on the new server, not just the homepages. Twenty sites failing the same way is one problem; catching it after three is a fifteen-minute fix, catching it after twenty is a rollback.</p>



<p class="wp-block-paragraph">The per-site sequence inside a batch:</p>



<ol class="wp-block-list">
<li>Drop the TTL on that domain&#8217;s records to a low value, at least a day ahead.</li>
<li>Run the delta rsync.</li>
<li>Dump, transfer and import the database.</li>
<li>Run search-replace if the URL is changing at all.</li>
<li>Test through a hosts-file entry or a resolved request, before DNS moves.</li>
<li>Change the A and AAAA records.</li>
<li>Put the old copy read-only.</li>
<li>Verify from outside, then move to the next site.</li>
</ol>



<p class="wp-block-paragraph">Step five is the one people skip and should not. You can send a request to the new server as if DNS had already changed, without changing anything:</p>



<pre class="wp-block-code"><code>curl -sSI --resolve example.com:443:NEW_SERVER_IP https://example.com/
curl -sS --resolve example.com:443:NEW_SERVER_IP https://example.com/wp-login.php | head -n 20</code></pre>



<p class="wp-block-paragraph">That gives you real status codes, real redirects and a real TLS handshake against the new box while the live site is untouched. If that returns a 301 loop or a 500, you have found it before any visitor did.</p>



<h2 class="wp-block-heading">Cron, email, and the things that run twice</h2>



<p class="wp-block-paragraph">During the overlap window you have two live copies of every site. Both have a database. Both have a scheduler.</p>



<p class="wp-block-paragraph">WordPress schedules through <code>wp-cron.php</code>, fired by visitor traffic. The old server still gets traffic, so it still fires cron, so it still sends the newsletter, still processes the subscription renewal, still runs the backup plugin. Subscribers get things twice. Worse, the site&#8217;s system crontab came across in your file sync and is now also running on the new box.</p>



<p class="wp-block-paragraph">Handle it explicitly. On the old copy, disable the loopback scheduler when you make it read-only:</p>



<pre class="wp-block-code"><code>define( 'DISABLE_WP_CRON', true );</code></pre>



<p class="wp-block-paragraph">And on the new server, set up a real system cron per site rather than leaving it to traffic, because a site with no traffic yet has no scheduler at all:</p>



<pre class="wp-block-code"><code>wp --path=/home/user/domains/example.com/public_html cron event run --due-now</code></pre>



<p class="wp-block-paragraph">Wrap that in a cron entry or a systemd timer at whatever interval the site needs, and check <code>wp cron event list</code> before assuming nothing important is queued.</p>



<p class="wp-block-paragraph">Email is the parallel problem and it fails quietly in the other direction. Your sites now send from a new IP. If SPF still authorizes only the old server&#8217;s address, and DKIM keys did not come across, and DMARC is set to anything stricter than none, your transactional mail starts landing in spam. Nothing bounces visibly. Password resets just stop arriving and you hear about it in support tickets a week later.</p>



<p class="wp-block-paragraph">Three records to check per domain. MX is the one not to touch casually: if mail lives elsewhere and you replace a zone wholesale, the mailboxes go with it.</p>



<ul class="wp-block-list">
<li><strong>MX:</strong> unchanged unless mail is genuinely moving with you.</li>
<li><strong>SPF:</strong> add the new server&#8217;s IP before cutover, remove the old one after the overlap window closes, not during.</li>
<li><strong>DKIM:</strong> keys are per-server. Either copy them or regenerate and publish the new public key.</li>
</ul>



<p class="wp-block-paragraph">The more resilient answer is to stop sending through the server at all. Routing WordPress mail through an authenticated SMTP relay means outbound deliverability stops depending on which machine the site happens to live on, which makes the next migration a non-event.</p>



<h2 class="wp-block-heading">Troubleshooting the weekend</h2>



<p class="wp-block-paragraph"><strong>Redirect loop on the new server.</strong> Usually <code>siteurl</code> and <code>home</code> disagreeing with what the vhost or reverse proxy is doing about HTTPS. Check both values with <code>wp option get siteurl</code> and <code>wp option get home</code>. If the site sits behind a proxy or CDN that terminates TLS, WordPress may see HTTP on the back end and redirect forever.</p>



<p class="wp-block-paragraph"><strong>Site loads, media does not.</strong> Either uploads did not finish syncing, ownership is wrong, or database paths still reference the old host. Compare a file listing on both sides before assuming it is a database problem.</p>



<p class="wp-block-paragraph"><strong>Widgets or theme options empty after import.</strong> This is the serialization damage described earlier. It is not recoverable by re-running the replacement. Restore the database from the dump and redo the URL change with WP-CLI.</p>



<p class="wp-block-paragraph"><strong>Certificate issuance failing repeatedly.</strong> Confirm what the challenge is actually hitting before retrying, because each failure spends part of your hourly authorization budget.</p>



<pre class="wp-block-code"><code>dig +noall +answer example.com A
dig +noall +answer @1.1.1.1 example.com A
openssl s_client -connect NEW_SERVER_IP:443 -servername example.com &lt; /dev/null 2&gt;/dev/null | openssl x509 -noout -subject -dates</code></pre>



<p class="wp-block-paragraph">The first query uses your local resolver, the second asks a public one directly, and the difference between them tells you whether you are looking at a real propagation state or your own DNS cache.</p>



<p class="wp-block-paragraph"><strong>Site fine, admin slow.</strong> Often a plugin trying to reach an external service that whitelisted the old IP. Look for API integrations, license checks and payment gateways in the manifest&#8217;s awkward-ones column.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Lowering TTL on cutover day. The old value is still cached; the change has no effect for as long as the old TTL says.</li>
<li>Cancelling the old server as soon as the sites load. Keep it running, read-only, for the whole overlap plus a margin. It is your rollback.</li>
<li>Replacing an entire DNS zone instead of editing the A record, and taking MX, SPF and verification TXT records with it.</li>
<li>Running search-replace before importing, or on the SQL file instead of the live database.</li>
<li>Testing only the homepage. Homepages are static and cached. Test a login, a form submission, an admin page and an uploaded image.</li>
<li>Leaving both schedulers live and finding out through duplicate customer emails.</li>
<li>Assuming the new server&#8217;s PHP extension set matches the old one because the version number does.</li>
<li>Treating rollback as a fleet-level decision. Per-site rollback is what keeps one bad site from stalling nineteen good ones.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a fleet move</h2>



<ul class="wp-block-list">
<li>Write the manifest first and update it as you go. It is the migration record and the handover document.</li>
<li>Take a restorable backup of each site to somewhere off both servers before you start, and test restoring one. A backup you have not restored is a hypothesis.</li>
<li>Warm sync early, delta sync at cutover, dump databases last. Batch by risk, and prove the stack on sites nobody will miss.</li>
<li>Verify every site from outside with a resolved request before its DNS moves.</li>
<li>Point uptime monitoring at the new IP before cutover, not after. UptimeRobot or Better Stack will tell you about a broken site faster than the client will.</li>
<li>Make the old copy read-only at the moment of DNS change, not at the end of the weekend.</li>
<li>Keep a rollback note per site: which records changed, what they were before, who can revert them.</li>
<li>Re-check cache configuration after the move rather than assuming it carried over, whether that is WP Rocket, WP Fastest Cache or a server-level layer.</li>
<li>Retire the old server on a scheduled date with a final archived backup, not by forgetting to renew it.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a bulk WordPress migration of 20 sites realistic in one weekend?</h3>



<p class="wp-block-paragraph">Yes, if the preparation happens in the week before. The weekend itself should be delta syncs, database moves and DNS changes. If you are still installing PHP extensions on Saturday morning, the weekend is not long enough.</p>



<h3 class="wp-block-heading">How low should I set the TTL, and how far ahead?</h3>



<p class="wp-block-paragraph">A low value like 300 seconds, set at least 24 to 48 hours before cutover so the previous TTL has fully expired everywhere. Going much lower than 300 buys little and increases query load. Raise it back after the migration settles.</p>



<h3 class="wp-block-heading">Will moving hosts hurt my search rankings?</h3>



<p class="wp-block-paragraph">Changing the server behind an unchanged domain is not itself a ranking event. What hurts is what goes wrong during it: 5xx responses while crawlers visit, failed certificates, redirect loops, a staging copy left indexable. Keep URLs identical, keep the site responding, and check Search Console afterwards.</p>



<h3 class="wp-block-heading">Can I use a migration plugin instead of the command line for this?</h3>



<p class="wp-block-paragraph">For one site, plugins are a reasonable choice and handle serialized data correctly. At twenty the arithmetic changes: twenty manual export and import cycles through a browser, with no dry run and no delta sync. The command line is repeatable and scriptable, which is the point at fleet scale.</p>



<h3 class="wp-block-heading">How long should I keep the old VPS running?</h3>



<p class="wp-block-paragraph">At least a week after the last site cuts over, in read-only mode. Some problems only appear over a full traffic cycle: weekly scheduled jobs, integrations that call in periodically, mail arriving at the old MX. Keep an archived backup after you destroy it.</p>



<h3 class="wp-block-heading">What if one site breaks after cutover?</h3>



<p class="wp-block-paragraph">Revert that site&#8217;s A record and lift the read-only restriction on the old copy. That is why the old server stays up and why rollback is per site. If you sync the database back afterwards, be careful not to overwrite writes that landed on the new server in the meantime.</p>



<h3 class="wp-block-heading">Do I need to move email at the same time?</h3>



<p class="wp-block-paragraph">No, and separating them is usually wiser. Move web first, confirm it is stable, then plan mailbox migration as its own change with its own window. Combining them means a single mistake in one DNS zone takes down both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A bulk WordPress migration is not twenty small migrations. It is one process run twenty times, and the difference is that at scale the visible failures get caught while the invisible ones do not.</p>



<p class="wp-block-paragraph">The copy is the easy part. rsync and WP-CLI are reliable, well documented and boring. What costs you is the overlap window: two servers both answering, both writing, both scheduling, for hours after you have mentally finished. Decide before the weekend what happens to writes on the old server, make that decision per site, and act on it at the moment DNS changes rather than at the end.</p>



<p class="wp-block-paragraph">Do that, batch by risk, and keep a rollback that works at the level of a single site, and twenty sites in a weekend is an ordinary piece of work rather than a gamble.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help moving a fleet of sites?</h2>



<p class="wp-block-paragraph">Migrations are a good thing to have a second pair of hands on, because the expensive mistakes are the quiet ones. This is work I do regularly:</p>



<ul class="wp-block-list">
<li>Auditing an existing fleet and producing the migration manifest, batch order and rollback plan before anything moves</li>
<li>Provisioning and hardening the destination VPS, including multiple PHP versions, control panel setup and firewall rules</li>
<li>Running the sync and database cutover with WP-CLI, including URL replacement on serialized data</li>
<li>Planning TLS issuance around ACME rate limits, including DNS-01 validation and automated renewal</li>
<li>Sorting out post-migration email: SPF, DKIM, DMARC and moving transactional mail onto an authenticated relay</li>
<li>Sitting in as the on-call engineer for the cutover window itself, or reviewing a plan you have already written</li>
</ul>



<p class="wp-block-paragraph">If you are planning a move and something in it worries you, send me the concrete thing: the site list, the current vhost config, the rsync output, the certbot error. I would rather look at the actual output than talk in general terms.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</title>
		<link>https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Ansible Collections]]></category>
		<category><![CDATA[Ansible Vault]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=204</guid>

					<description><![CDATA[<p>A single Ansible run can take a blank VPS to a hardened, reproducible server. The hard part is not the tasks, it is the order. Here is the ordering that keeps you logged in, the failures that pass green and bite two weeks later, and how to make the second run as safe as the first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.</p>



<p class="wp-block-paragraph">That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report <code>changed</code>, do nothing useful, and leave you believing the server is hardened when it is not.</p>



<p class="wp-block-paragraph">This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.</p>



<h2 class="wp-block-heading">What a one-run bootstrap actually has to survive</h2>



<p class="wp-block-paragraph">A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.</p>



<p class="wp-block-paragraph">Four things break it, and they break it in different ways:</p>



<ul class="wp-block-list">
<li><strong>The chicken-and-egg problem.</strong> Ansible needs Python on the target to run almost any module, and a minimal image may not have it.</li>

<li><strong>Lockout.</strong> You remove your own access before proving the replacement access works.</li>

<li><strong>Non-idempotent tasks.</strong> The first run is fine. The second run is where the damage happens.</li>

<li><strong>Missing collections.</strong> The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with <code>ansible-core</code>.</li>
</ul>



<p class="wp-block-paragraph">Everything below is organised around those four, because that is how they actually show up.</p>



<h2 class="wp-block-heading">The chicken-and-egg problem: no Python, no modules</h2>



<p class="wp-block-paragraph">Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.</p>



<p class="wp-block-paragraph">The escape hatch is <code>ansible.builtin.raw</code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with <code>gather_facts: false</code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.</p>



<pre class="wp-block-code"><code>- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update &amp;&amp; apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:</code></pre>



<p class="wp-block-paragraph">Two details worth understanding rather than copying. <code>failed_when: false</code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And <code>ansible.builtin.setup</code> is the fact-gathering module called explicitly, which means from that point on <code>ansible_distribution</code> and friends are available even though the play started with gathering disabled.</p>



<p class="wp-block-paragraph">If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.</p>



<h2 class="wp-block-heading">Failure family one: locking yourself out while the playbook says ok</h2>



<p class="wp-block-paragraph">This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.</p>



<h3 class="wp-block-heading">The order that survives</h3>



<p class="wp-block-paragraph">The rule is simple: <strong>nothing that removes access happens until something has proven the new access works.</strong> In practice:</p>



<ol class="wp-block-list">
<li>Connect as whatever the provider handed you, usually <code>root</code> or a sudo-capable default user.</li>

<li>Create the admin account, add it to the sudo or wheel group, install its authorized key.</li>

<li>Write the sudoers drop-in and validate it with <code>visudo -cf</code>.</li>

<li><strong>Prove you can log in as that account and escalate.</strong> This is the step everyone skips.</li>

<li>Install the firewall and allow SSH <em>before</em> enabling it.</li>

<li>Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.</li>

<li>Only now disable root login and password authentication.</li>
</ol>



<p class="wp-block-paragraph">Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:</p>



<pre class="wp-block-code"><code>- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"</code></pre>



<p class="wp-block-paragraph">Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.</p>



<h3 class="wp-block-heading">The socket activation trap</h3>



<p class="wp-block-paragraph">This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. <code>ssh.socket</code> owns the listening port, and <code>Port</code> in <code>sshd_config</code> is not what decides where the daemon listens.</p>



<p class="wp-block-paragraph">So your playbook writes <code>Port 2222</code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.</p>



<p class="wp-block-paragraph">The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:</p>



<pre class="wp-block-code"><code># Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'</code></pre>



<p class="wp-block-paragraph">If <code>ssh.socket</code> is enabled, the override needs an empty <code>ListenStream=</code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.</p>



<p class="wp-block-paragraph">Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.</p>



<h3 class="wp-block-heading">Firewall ordering</h3>



<p class="wp-block-paragraph">UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an <code>UNREACHABLE</code> that will not recover. Rules go in first, always:</p>



<pre class="wp-block-code"><code>- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled</code></pre>



<p class="wp-block-paragraph">Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.</p>



<h2 class="wp-block-heading">Failure family two: the second run is the dangerous one</h2>



<p class="wp-block-paragraph">A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are <code>shell</code> and <code>command</code> tasks with no guard.</p>



<p class="wp-block-paragraph">An unguarded <code>shell</code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.</p>



<p class="wp-block-paragraph">Guard them properly:</p>



<ul class="wp-block-list">
<li><code>creates:</code> or <code>removes:</code> makes the task skip when the target state already exists. Prefer this.</li>

<li><code>changed_when: false</code> for read-only commands so they stop lying about changing things.</li>

<li><code>when:</code> driven by a registered check, for anything conditional.</li>
</ul>



<p class="wp-block-paragraph">The other idempotency trap is <code>lineinfile</code> against <code>sshd_config</code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:</p>



<pre class="wp-block-code"><code>- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false</code></pre>



<p class="wp-block-paragraph">Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so <code>validate:</code> on the template task with <code>sshd -t -f %s</code> will fail on a temporary file that has no host keys and no context. Run <code>sshd -t</code> as its own task afterwards instead, and let the handler restart only if it passes.</p>



<p class="wp-block-paragraph">Two prerequisites for the drop-in approach. The base <code>sshd_config</code> needs an <code>Include /etc/ssh/sshd_config.d/*.conf</code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the <em>first</em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.</p>



<h2 class="wp-block-heading">Failure family three: the collections are not where you think</h2>



<p class="wp-block-paragraph"><code>ansible-core</code> ships the <code>ansible.builtin</code> namespace and essentially nothing else. The full <code>ansible</code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.</p>



<p class="wp-block-paragraph">Things people assume are built in and are not: <code>ufw</code> and <code>timezone</code> live in <code>community.general</code>. <code>authorized_key</code>, <code>sysctl</code> and <code>mount</code> live in <code>ansible.posix</code>. Declare them and pin them:</p>



<pre class="wp-block-code"><code># requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml</code></pre>



<p class="wp-block-paragraph">Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. <code>ansible-lint</code> flags this and it is worth listening to.</p>



<h2 class="wp-block-heading">Failure family four: secrets in the repository</h2>



<p class="wp-block-paragraph">Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.</p>



<p class="wp-block-paragraph">Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:</p>



<pre class="wp-block-code"><code>ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass</code></pre>



<p class="wp-block-paragraph">The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Putting it together: how to bootstrap a VPS with Ansible end to end</h2>



<p class="wp-block-paragraph">Structure matters less than order, but a shape that works well is a single <code>site.yml</code> calling roles in dependency order, with the risky access changes fenced off at the end.</p>



<pre class="wp-block-code"><code># inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22</code></pre>



<p class="wp-block-paragraph">Then the run itself, in this order:</p>



<ol class="wp-block-list">
<li><strong>Reachability.</strong> Python check, fact gathering, timezone, hostname, package cache refresh.</li>

<li><strong>Identity.</strong> Admin user, group membership, authorized key, sudoers drop-in with <code>visudo -cf</code> validation.</li>

<li><strong>Verification gate.</strong> The task shown earlier. Nothing destructive runs before this passes.</li>

<li><strong>Firewall.</strong> Rules first, policy second, enable third.</li>

<li><strong>Unattended updates.</strong> <code>unattended-upgrades</code> on Debian and Ubuntu, <code>dnf-automatic</code> with its systemd timer on RHEL-family hosts.</li>

<li><strong>Brute-force protection.</strong> fail2ban with your own <code>/etc/fail2ban/jail.local</code>, never edits to <code>jail.conf</code>.</li>

<li><strong>Workload.</strong> Docker, a web server, a monitoring agent, whatever the box is for.</li>

<li><strong>Access teardown.</strong> Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.</li>
</ol>



<p class="wp-block-paragraph">When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:</p>



<pre class="wp-block-code"><code>- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60</code></pre>



<p class="wp-block-paragraph"><code>meta: reset_connection</code> does not accept a <code>when</code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.</p>



<h2 class="wp-block-heading">Test it before you point it at anything real</h2>



<p class="wp-block-paragraph">The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.</p>



<pre class="wp-block-code"><code># catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff</code></pre>



<p class="wp-block-paragraph">One caveat on <code>--check</code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong><code>UNREACHABLE</code> on the very first task.</strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with <code>ssh-keygen -R</code> and re-add the correct one with <code>ssh-keyscan</code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.</li>

<li><strong><code>UNREACHABLE</code> immediately after a firewall task.</strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.</li>

<li><strong>&#8220;Missing sudo password&#8221;.</strong> The sudoers drop-in is not in effect or has bad syntax. Validate with <code>visudo -cf</code> on the file itself before trusting it.</li>

<li><strong>Module not found under an FQCN.</strong> The collection is not installed on the machine running the playbook. Confirm with <code>ansible-galaxy collection list</code>.</li>

<li><strong>New group membership ignored.</strong> Adding a user to <code>docker</code> mid-play does not affect the already-open connection. Insert <code>meta: reset_connection</code> after the group change.</li>

<li><strong>fail2ban installed but not banning.</strong> On images without rsyslog, <code>/var/log/auth.log</code> may not exist, so a jail configured to read it never starts. Check <code>fail2ban-client status</code> and switch the jail to the systemd backend if the log file is absent.</li>

<li><strong>Everything green, port unchanged.</strong> Socket activation. Check <code>systemctl is-enabled ssh.socket</code> and <code>ss -tlnp</code>.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling root login in the same play that creates the replacement user, with no verification step between them.</li>

<li>Enabling UFW before the SSH allow rule exists.</li>

<li>Assuming <code>Port</code> in <code>sshd_config</code> is authoritative on every distribution.</li>

<li>Unguarded <code>shell</code> tasks that notify handlers, causing service restarts on every run.</li>

<li>Editing <code>jail.conf</code> instead of creating <code>jail.local</code>, so the next package update silently reverts it.</li>

<li>Committing plaintext secrets because &#8220;it is only a bootstrap playbook&#8221;.</li>

<li>Testing on a host you cannot afford to lose, with no console access configured.</li>

<li>Relying on <code>--check</code> as a substitute for a real run on a disposable host.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.</li>

<li>Put a verification task before every irreversible one, and let the play fail there rather than after.</li>

<li>Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.</li>

<li>Validate configuration before restarting the service that reads it. <code>sshd -t</code>, <code>visudo -cf</code>, <code>nginx -t</code>.</li>

<li>Pin collections in <code>requirements.yml</code> and install them in CI, not just locally.</li>

<li>Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.</li>

<li>Reboot once during testing. Half of the silent failures only appear after a restart.</li>

<li>Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you really bootstrap a VPS with Ansible in a single playbook run?</h3>



<p class="wp-block-paragraph">Yes, including changing the SSH user and port mid-run. The mechanism is <code>set_fact</code> to update the connection variables, <code>meta: reset_connection</code> to drop the cached connection, and <code>wait_for_connection</code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.</p>



<h3 class="wp-block-heading">Should I use cloud-init or Ansible for initial server setup?</h3>



<p class="wp-block-paragraph">Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.</p>



<h3 class="wp-block-heading">Do I need Terraform or OpenTofu as well?</h3>



<p class="wp-block-paragraph">Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.</p>



<h3 class="wp-block-heading">Why does my playbook work the first time and fail the second?</h3>



<p class="wp-block-paragraph">Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.</p>



<h3 class="wp-block-heading">Is changing the SSH port worth doing?</h3>



<p class="wp-block-paragraph">It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.</p>



<h3 class="wp-block-heading">How do I recover if the playbook locks me out?</h3>



<p class="wp-block-paragraph">Through the provider&#8217;s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.</p>



<h3 class="wp-block-heading">Can this run against several servers at once?</h3>



<p class="wp-block-paragraph">Yes, but use <code>serial: 1</code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.</p>



<p class="wp-block-paragraph">Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a bootstrap playbook you can actually rerun?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:</p>



<ul class="wp-block-list">
<li>Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.</li>

<li>Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.</li>

<li>Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.</li>

<li>Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.</li>

<li>Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.</li>

<li>Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.</li>
</ul>



<p class="wp-block-paragraph">Send me the playbook, a recap from a second run, or the output of <code>ss -tlnp</code> from the host that is behaving oddly, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Automating Server Provisioning for New Clients Without Building a Snowflake Farm</title>
		<link>https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Freelancing]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=192</guid>

					<description><![CDATA[<p>Hand-built client servers fail quietly: you only find out they were never reproducible on the day you need to rebuild one. Here is how to automate server provisioning for new clients using cloud-init for first boot, Ansible for convergence, per-client secrets and a verification gate that refuses to hand over a half-built box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email is usually short. &#8220;Can you spin up a staging box that matches production?&#8221; You open your notes from eight months ago, find a half-finished shell history and a text file called <code>setup-notes-final2.txt</code>, and the honest answer is no. You can build something close. You cannot build the same machine.</p>



<p class="wp-block-paragraph">That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between &#8220;the server works&#8221; and &#8220;the server is reproducible&#8221; turns into a weekend.</p>



<p class="wp-block-paragraph">This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client&#8217;s credentials from ever touching another client&#8217;s box, and how to prove a build is finished instead of assuming it.</p>



<h2 class="wp-block-heading">The four failure families worth designing against</h2>



<p class="wp-block-paragraph">Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.</p>



<h3 class="wp-block-heading">1. You cannot rebuild it</h3>



<p class="wp-block-paragraph">The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run <code>apt install</code>, every file edited over SSH at 1am, every &#8220;I&#8217;ll document that later&#8221; is a piece of the machine that exists nowhere but the machine.</p>



<p class="wp-block-paragraph">This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.</p>



<h3 class="wp-block-heading">2. Every client is a slightly different server</h3>



<p class="wp-block-paragraph">Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.</p>



<p class="wp-block-paragraph">Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts &#8220;all our clients have&#8230;&#8221; becomes a guess.</p>



<h3 class="wp-block-heading">3. One credential, too much blast radius</h3>



<p class="wp-block-paragraph">This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client&#8217;s server, and the same API token provisions every client&#8217;s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.</p>



<p class="wp-block-paragraph">Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.</p>



<h3 class="wp-block-heading">4. The handover you cannot perform</h3>



<p class="wp-block-paragraph">Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How to automate server provisioning for new clients</h2>



<p class="wp-block-paragraph">The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.</p>



<h3 class="wp-block-heading">Layer 1: the machine itself</h3>



<p class="wp-block-paragraph">Terraform or OpenTofu talks to your provider&#8217;s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client&#8217;s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.</p>



<p class="wp-block-paragraph">Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.</p>



<h3 class="wp-block-heading">Layer 2: first boot with cloud-init</h3>



<p class="wp-block-paragraph">cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.</p>



<pre class="wp-block-code"><code>#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo</code></pre>



<p class="wp-block-paragraph">Two deliberate omissions. There is no <code>package_upgrade: true</code>, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in <code>runcmd</code> is a script you will never test again.</p>



<p class="wp-block-paragraph"><code>python3</code> is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.</p>



<p class="wp-block-paragraph">Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:</p>



<pre class="wp-block-code"><code>cloud-init schema --config-file user-data.yaml</code></pre>



<h3 class="wp-block-heading">Layer 3: convergence with Ansible</h3>



<p class="wp-block-paragraph">This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.</p>



<p class="wp-block-paragraph">The layout that scales across clients separates the shared build from the per-client differences:</p>



<pre class="wp-block-code"><code>infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml</code></pre>



<p class="wp-block-paragraph">Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.</p>



<p class="wp-block-paragraph">Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:</p>



<pre class="wp-block-code"><code>cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  &gt; ../inventory/hosts.ini</code></pre>



<p class="wp-block-paragraph">There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.</p>



<h3 class="wp-block-heading">Layer 4: the verification gate</h3>



<p class="wp-block-paragraph">This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.</p>



<p class="wp-block-paragraph">Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:</p>



<pre class="wp-block-code"><code>- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost</code></pre>



<p class="wp-block-paragraph">Note <code>ssh_unit</code> as a variable rather than a literal: the SSH unit is <code>ssh.service</code> on Debian and Ubuntu but <code>sshd.service</code> on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.</p>



<p class="wp-block-paragraph">The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.</p>



<h2 class="wp-block-heading">Keeping clients apart: secrets and access</h2>



<p class="wp-block-paragraph">Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client&#8217;s infrastructure is a single, contained action.</p>



<p class="wp-block-paragraph">For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.</p>



<pre class="wp-block-code"><code># .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...</code></pre>



<p class="wp-block-paragraph">Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.</p>



<p class="wp-block-paragraph">For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.</p>



<p class="wp-block-paragraph">If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.</p>



<h2 class="wp-block-heading">Troubleshooting a provisioning run</h2>



<h3 class="wp-block-heading">Ansible connects before cloud-init has finished</h3>



<p class="wp-block-paragraph">The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.</p>



<p class="wp-block-paragraph">Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:</p>



<pre class="wp-block-code"><code>cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log</code></pre>



<p class="wp-block-paragraph">One caveat that catches people: never put <code>cloud-init status --wait</code> inside a <code>runcmd</code> or <code>bootcmd</code> block. It will deadlock waiting for the process that is running it.</p>



<h3 class="wp-block-heading">cloud-init appears to have done nothing</h3>



<p class="wp-block-paragraph">Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in <code>/var/lib/cloud/instance/sem/</code> are preventing a rerun.</p>



<pre class="wp-block-code"><code># what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot</code></pre>



<p class="wp-block-paragraph">Never run <code>cloud-init clean</code> on a production host. It removes cached state and the following boot will behave as a first boot.</p>



<h3 class="wp-block-heading">Host key verification failed on a rebuilt server</h3>



<p class="wp-block-paragraph">Rebuild a machine on the same IP and every client with the old key in <code>known_hosts</code> refuses to connect. The tempting fix is to disable host key checking in <code>ansible.cfg</code>. Do not. That turns a real integrity check into a permanent no-op across every client you manage.</p>



<p class="wp-block-paragraph">Remove the stale entry and pin the new one as part of the provisioning flow:</p>



<pre class="wp-block-code"><code>ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 &gt;&gt; ~/.ssh/known_hosts</code></pre>



<p class="wp-block-paragraph">Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.</p>



<h3 class="wp-block-heading">The playbook reports changes on every single run</h3>



<p class="wp-block-paragraph">Almost always a <code>shell</code> or <code>command</code> task with no <code>creates</code>, <code>changed_when</code> or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.</p>



<p class="wp-block-paragraph">Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:</p>



<pre class="wp-block-code"><code>ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Putting the application build in <code>runcmd</code>. It runs once, never gets tested, and is invisible to every later change.</li>

<li>One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.</li>

<li>Copying a role into a client directory to make one small change, instead of extracting a variable.</li>

<li>Treating &#8220;the playbook exited zero&#8221; as proof the server is ready. It is proof no task threw.</li>

<li>Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.</li>

<li>Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.</li>

<li>Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Rebuild before you need to.</strong> Destroy and rebuild a client&#8217;s staging host on a schedule. It is the only honest test of reproducibility.</li>

<li><strong>Version your roles and pin them per client.</strong> A shared role that everyone tracks on main means one commit changes six clients at once.</li>

<li><strong>Keep first boot minimal.</strong> If it can wait for Ansible, it should.</li>

<li><strong>Make monitoring part of provisioning.</strong> Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.</li>

<li><strong>Same for backups.</strong> An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.</li>

<li><strong>Separate credentials per client, always.</strong> Keys, tokens, encryption keys, state backends.</li>

<li><strong>Write the runbook as you build.</strong> The provisioning repo&#8217;s README is the handover document. If it is accurate, the engagement can end cleanly.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Terraform if I only manage a few VPS instances?</h3>



<p class="wp-block-paragraph">Not necessarily. If you are creating three servers a year through a provider&#8217;s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.</p>



<h3 class="wp-block-heading">Should I build custom images instead of provisioning at boot?</h3>



<p class="wp-block-paragraph">Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.</p>



<h3 class="wp-block-heading">Terraform or OpenTofu for client projects?</h3>



<p class="wp-block-paragraph">Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.</p>



<h3 class="wp-block-heading">How do I automate server provisioning without exposing SSH to the internet?</h3>



<p class="wp-block-paragraph">The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.</p>



<h3 class="wp-block-heading">How do I stop configuration drift once the server is live?</h3>



<p class="wp-block-paragraph">Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.</p>



<h3 class="wp-block-heading">What belongs in cloud-init versus Ansible?</h3>



<p class="wp-block-paragraph">cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.</p>



<h3 class="wp-block-heading">How long should provisioning a new client server take?</h3>



<p class="wp-block-paragraph">Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer &#8220;can you rebuild this?&#8221; with yes, on any day, without opening a shell on the existing machine.</p>



<p class="wp-block-paragraph">Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.</p>



<p class="wp-block-paragraph">Start with one client. Get their staging box rebuilding from scratch. Then make the second client&#8217;s build reuse the roles from the first. That is the whole path.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a provisioning setup you can actually hand over?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:</p>



<ul class="wp-block-list">
<li>Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime</li>

<li>Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets</li>

<li>Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works</li>

<li>Writing the verification playbook that decides whether a build is finished, and wiring it into CI</li>

<li>Fixing idempotency so check-mode runs become a usable drift report instead of noise</li>

<li>Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
