<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Let&#039;s Encrypt | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/lets-encrypt/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/lets-encrypt/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 13:16:30 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Let&#039;s Encrypt | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/lets-encrypt/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</title>
		<link>https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Decommissioning]]></category>
		<category><![CDATA[DirectAdmin]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[search-replace]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SPF]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Website Migration]]></category>
		<category><![CDATA[WordPress Hosting]]></category>
		<category><![CDATA[WordPress Migration]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=301</guid>

					<description><![CDATA[<p>Moving one WordPress site is easy. Moving twenty in a weekend is a pipeline, and the failures that cost you are the quiet ones: writes landing on the old server, cron firing on two boxes, certificates blocked by ACME rate limits. Here is the fleet migration process I use, from manifest to per-site rollback.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message arrived on Monday morning. Three quote requests submitted over the weekend, confirmed by the people who submitted them, and none of them in the CRM. The site was up. The contact form worked when I tested it. Nothing in the error log.</p>



<p class="wp-block-paragraph">Those submissions went to the old server. It was still answering for a slice of the internet twenty hours after the DNS change, and by then I had already stopped syncing it.</p>



<p class="wp-block-paragraph">That is the failure that separates a bulk WordPress migration from moving a single site. When you move one site you watch it. When you move twenty in a weekend, you are running a pipeline, and the things that go wrong are the ones nobody is watching: writes landing on a server you have mentally decommissioned, cron firing on two boxes at once, a certificate that never issued because a DNS record was still pointing at the old IP when the challenge ran.</p>



<p class="wp-block-paragraph">This is the process I use for fleet moves onto a single VPS: the manifest that makes the weekend repeatable, the two-pass sync, the database step people rush, the certificate limits that bite specifically at fleet scale, and a cutover order that keeps rollback cheap. It assumes SSH on both boxes and some comfort on the command line.</p>



<h2 class="wp-block-heading">The failure that actually costs you: writes landing on the old server</h2>



<p class="wp-block-paragraph">Every migration guide tells you to lower your TTL. Almost none of them tell you what happens in the gap afterwards, and the gap is where the money goes.</p>



<p class="wp-block-paragraph">Here is the mechanism. You change the A record. Most resolvers pick up the new IP inside the TTL window. Some do not: corporate resolvers that clamp minimum TTLs, ISP resolvers that ignore short values, browsers holding a connection open, a mail server that resolved the host an hour ago. For those clients, the old server is still the site. It still has a working database. It still accepts comments, form entries, orders, user registrations and password resets.</p>



<p class="wp-block-paragraph">Nothing errors. Nothing logs. The write succeeds on a database you are about to throw away.</p>



<p class="wp-block-paragraph">There are only two honest answers to this, and picking one per site is the first real decision of the weekend.</p>



<ul class="wp-block-list">
<li><strong>Accept the loss.</strong> Fine for brochure sites, documentation, most blogs. A missed comment is not a business problem.</li>
<li><strong>Make the old server read-only the moment DNS changes.</strong> Mandatory for anything with a checkout, a booking flow, a members area or a lead form that feeds a sales process.</li>
</ul>



<p class="wp-block-paragraph">Read-only in practice means maintenance mode on the old copy, a static holding page on the old vhost, or pointing the old install&#8217;s <code>wp-config.php</code> at a database user with SELECT only. Any of those turns silent data loss into a visible error the visitor can react to. Someone who sees &#8220;temporarily unavailable&#8221; comes back. Someone whose order vanished does not.</p>



<p class="wp-block-paragraph">Do not skip this because the old server &#8220;will be off soon&#8221;. The window between DNS change and old server shutdown is the longest, least supervised part of the whole weekend.</p>



<h2 class="wp-block-heading">Build the fleet manifest before you touch a server</h2>



<p class="wp-block-paragraph">Twenty sites is the point where memory stops working. You will not remember which one runs an old PHP branch, which one has a hardcoded absolute path in a theme, which one has MX records pointing somewhere else entirely.</p>



<p class="wp-block-paragraph">So the first artifact is a table. One row per site, filled in before anything moves.</p>



<ul class="wp-block-list">
<li>Document root and database name</li>
<li>Current <code>siteurl</code> and <code>home</code> values, exactly as stored</li>
<li>PHP version the site currently runs under</li>
<li>Disk size of the document root and dump size of the database</li>
<li>Where DNS is authoritative, and who holds the login</li>
<li>Whether mail for that domain is on this server or elsewhere</li>
<li>Does it take writes from the public: comments, forms, checkout, registration</li>
<li>Cache and object cache in use</li>
<li>Cutover batch number and rollback owner</li>
</ul>



<p class="wp-block-paragraph">Most of that you can pull off the old server rather than typing it. This walks a DirectAdmin-style layout and prints a tab-separated line per WordPress install:</p>



<pre class="wp-block-code"><code>for d in /home/*/domains/*/public_html; do
  [ -f "$d/wp-config.php" ] || continue
  printf '%st%st%sn' 
    "$d" 
    "$(wp --path="$d" option get siteurl --skip-plugins --skip-themes 2&gt;/dev/null)" 
    "$(du -sh "$d" | cut -f1)"
done</code></pre>



<p class="wp-block-paragraph">The <code>--skip-plugins --skip-themes</code> flags matter here. Without them WP-CLI boots the full site, and one broken plugin on one install will stop your inventory halfway through with a fatal error.</p>



<p class="wp-block-paragraph">The DNS column is the one people fill in last and regret. Check where each zone actually lives before the weekend, not during it. A domain whose nameservers point at Cloudflare while you are editing records in the old host&#8217;s DNS panel will not move no matter how many times you refresh.</p>



<h2 class="wp-block-heading">Rebuild the stack, don&#8217;t clone the disk</h2>



<p class="wp-block-paragraph">There is a tempting shortcut where you image the old server and restore it onto the new one. It usually works and it is usually the wrong call, because you have just carried across every accumulated fix, every orphaned config, and every reason the old box got slow enough to need replacing.</p>



<p class="wp-block-paragraph">Build the destination clean. Whatever the provider is, whether that is Contabo, InterServer, Hetzner or a machine you already have, the sequence is the same: base OS, web server, PHP with the version and extensions the manifest says you need, database server, control panel if you use one, firewall, then a hardening pass.</p>



<p class="wp-block-paragraph">Two things to get right before any files land.</p>



<p class="wp-block-paragraph"><strong>PHP versions.</strong> If three sites need an older branch and seventeen are happy on current, install both and make the per-site assignment explicit in the manifest. Discovering the mismatch after cutover means a white screen on a live domain instead of a note in a spreadsheet. Check the extension list too, not just the version number, because <code>imagick</code>, <code>soap</code> and <code>intl</code> are the usual absentees.</p>



<p class="wp-block-paragraph"><strong>Access during the weekend.</strong> If you lock SSH and the control panel behind an IP allowlist, decide beforehand what your source address will be. A rotating home IP is how people lock themselves out of their own migration at two in the morning. A dedicated-IP option from a VPN provider such as NordVPN or Surfshark gives you one stable address to allowlist; a second admin path on a non-standard port works too. Deciding at the moment you are locked out does not.</p>



<h2 class="wp-block-heading">Two-pass rsync beats one big copy</h2>



<p class="wp-block-paragraph">The mistake at fleet scale is treating the file copy as a cutover-day task. Copy everything on cutover day and your maintenance window is however long twenty document roots take, which for a media-heavy fleet is hours you do not have.</p>



<p class="wp-block-paragraph">Split it. Do a warm sync days in advance while both sites are live, then a delta sync at cutover that moves only what changed.</p>



<p class="wp-block-paragraph">Warm sync, run from the old server, no deletes:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph">What those flags buy you: <code>-a</code> preserves permissions, times, symlinks and ownership; <code>-H</code> keeps hard links, which some backup plugins and dedup schemes rely on; <code>-A</code> and <code>-X</code> carry ACLs and extended attributes; <code>--numeric-ids</code> stops rsync remapping ownership through name lookups when the UIDs differ between boxes, which is exactly what happens when the new server creates users in a different order.</p>



<p class="wp-block-paragraph">Delta sync at cutover, same command with one addition:</p>



<pre class="wp-block-code"><code>rsync -aHAX --numeric-ids --delete --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/</code></pre>



<p class="wp-block-paragraph"><code>--delete</code> removes files on the destination that no longer exist on the source. It is what makes the second pass an accurate mirror rather than a merge of two states. It is also the flag that will destroy your afternoon if you get the paths wrong, so check the trailing slashes: a source path ending in a slash copies the contents of the directory, one without it copies the directory itself, one level deeper than you meant.</p>



<p class="wp-block-paragraph">Run the delta pass once with <code>--dry-run</code> before running it for real. And on SELinux systems, relabel the tree afterwards: files arriving over rsync carry contexts the web server may not accept, and that produces a 403 which has nothing to do with permission bits.</p>



<h2 class="wp-block-heading">Databases last, and never with sed</h2>



<p class="wp-block-paragraph">Files can be stale for a few days. Databases cannot. Dump each database as late as you reasonably can, immediately before that site&#8217;s cutover, not on Friday for a Sunday move.</p>



<pre class="wp-block-code"><code># on the old server, per site
wp --path=/home/user/domains/example.com/public_html db export 
  /root/migration/example.com.sql --skip-plugins --skip-themes</code></pre>



<p class="wp-block-paragraph">Import on the new box, then handle URLs. If the domain is not changing, you may have nothing to replace. If the site is moving from HTTP to HTTPS, or from a staging hostname, or from a temporary preview domain you used for testing, you do.</p>



<p class="wp-block-paragraph">The rule that matters: do not run a find-and-replace over the SQL file with <code>sed</code>. WordPress stores widget settings, theme options, page builder layouts and plugin config as PHP serialized arrays, and every serialized string carries a byte-length prefix. Change the string without recalculating the prefix and the array fails to unserialize. The site does not error loudly. It loses its widgets, or its theme options revert to defaults, or the page builder renders an empty page.</p>



<p class="wp-block-paragraph">WP-CLI unserializes, replaces and reserializes, which is why it is the only tool I use for this.</p>



<pre class="wp-block-code"><code>wp search-replace 'http://example.com' 'https://example.com' 
  --all-tables-with-prefix 
  --precise 
  --skip-columns=guid 
  --report-changed-only 
  --dry-run</code></pre>



<p class="wp-block-paragraph">Read the dry-run output before you commit. Each flag earns its place: <code>--all-tables-with-prefix</code> catches plugin tables that share your prefix and would otherwise be skipped, <code>--precise</code> forces a full unserialize pass on every column instead of the faster string-level path, and <code>--skip-columns=guid</code> leaves the GUID column alone because feed readers use it as an identifier, not as a URL to fetch. Drop <code>--dry-run</code> to apply.</p>



<p class="wp-block-paragraph">Flush the object cache afterwards if the site uses one. A stale Redis or Memcached entry will happily serve you the old values and convince you the replacement failed.</p>



<h2 class="wp-block-heading">TLS at fleet scale: where Let&#8217;s Encrypt rate limits actually bite</h2>



<p class="wp-block-paragraph">Twenty certificates in one weekend is where a limit you have never noticed becomes the thing that blocks you.</p>



<p class="wp-block-paragraph">Let&#8217;s Encrypt publishes its limits and they are worth reading properly before a fleet move. Two of them are the ones that catch migrations.</p>



<ul class="wp-block-list">
<li><strong>New certificates per registered domain.</strong> Up to 50 certificates per registered domain every 7 days, counted globally across all accounts. If your twenty sites are twenty separate registered domains, you are nowhere near it. If they are subdomains of one or two parent domains, which is common for an agency fleet, you can burn through it.</li>
<li><strong>Authorization failures per identifier per account.</strong> Up to 5 failures per hostname per account per hour, refilling at one every 12 minutes. This is the one that actually bites, because it is triggered by the exact thing you are doing: requesting a certificate for a hostname whose DNS still points at the old server, so the HTTP-01 challenge is served by the wrong machine and fails.</li>
</ul>



<p class="wp-block-paragraph">Order matters as a result. If you issue before DNS moves, HTTP-01 fails. If you issue after DNS moves, visitors hit a certificate error in the gap. Three ways out, in the order I reach for them:</p>



<ol class="wp-block-list">
<li><strong>DNS-01 validation.</strong> Proves control through a TXT record instead of an HTTP request, so it does not care where the A record points. You can issue every certificate before touching DNS. This is the clean answer if your DNS provider has an API your ACME client supports.</li>
<li><strong>Cut DNS first, issue immediately after.</strong> Simple, and the exposure is short if you are watching. Batch it so you are never waiting on twenty challenges at once.</li>
<li><strong>Copy the existing certificates and keys across.</strong> Buys you a bridge until the new server can renew on its own. Only viable if you control the old server&#8217;s filesystem and the certificates have real time left.</li>
</ol>



<p class="wp-block-paragraph">Whichever you choose, test the plumbing against the staging environment first. Most ACME clients expose this as a dry-run mode, and it exists precisely so you can fail as many times as you like without spending production budget.</p>



<p class="wp-block-paragraph">One more thing that catches people: if any site has HSTS enabled with a long max-age, browsers that have seen it will refuse to fall back to HTTP. There is no grace period and no way to talk them out of it. Those sites need a valid certificate on the new server from the first request, which pushes them firmly toward DNS-01 or a copied certificate.</p>



<h2 class="wp-block-heading">Planning the cutover order for a bulk WordPress migration</h2>



<p class="wp-block-paragraph">Do not move twenty sites at once. Batch them, and batch by risk rather than alphabetically or by size.</p>



<ol class="wp-block-list">
<li><strong>Batch one: the throwaways.</strong> Two or three low-traffic sites you own or that nobody would notice. This batch exists to prove the stack, not to make progress.</li>
<li><strong>Batch two: the ordinary majority.</strong> Standard brochure and content sites, no checkout, no logins that matter.</li>
<li><strong>Batch three: the ones with writes.</strong> Stores, membership sites, anything with a form feeding a sales process.</li>
<li><strong>Batch four: the awkward ones.</strong> Custom stacks, odd PHP requirements, third-party integrations that whitelist your server IP.</li>
</ol>



<p class="wp-block-paragraph">Stop between batches. Look at the error logs on the new server, not just the homepages. Twenty sites failing the same way is one problem; catching it after three is a fifteen-minute fix, catching it after twenty is a rollback.</p>



<p class="wp-block-paragraph">The per-site sequence inside a batch:</p>



<ol class="wp-block-list">
<li>Drop the TTL on that domain&#8217;s records to a low value, at least a day ahead.</li>
<li>Run the delta rsync.</li>
<li>Dump, transfer and import the database.</li>
<li>Run search-replace if the URL is changing at all.</li>
<li>Test through a hosts-file entry or a resolved request, before DNS moves.</li>
<li>Change the A and AAAA records.</li>
<li>Put the old copy read-only.</li>
<li>Verify from outside, then move to the next site.</li>
</ol>



<p class="wp-block-paragraph">Step five is the one people skip and should not. You can send a request to the new server as if DNS had already changed, without changing anything:</p>



<pre class="wp-block-code"><code>curl -sSI --resolve example.com:443:NEW_SERVER_IP https://example.com/
curl -sS --resolve example.com:443:NEW_SERVER_IP https://example.com/wp-login.php | head -n 20</code></pre>



<p class="wp-block-paragraph">That gives you real status codes, real redirects and a real TLS handshake against the new box while the live site is untouched. If that returns a 301 loop or a 500, you have found it before any visitor did.</p>



<h2 class="wp-block-heading">Cron, email, and the things that run twice</h2>



<p class="wp-block-paragraph">During the overlap window you have two live copies of every site. Both have a database. Both have a scheduler.</p>



<p class="wp-block-paragraph">WordPress schedules through <code>wp-cron.php</code>, fired by visitor traffic. The old server still gets traffic, so it still fires cron, so it still sends the newsletter, still processes the subscription renewal, still runs the backup plugin. Subscribers get things twice. Worse, the site&#8217;s system crontab came across in your file sync and is now also running on the new box.</p>



<p class="wp-block-paragraph">Handle it explicitly. On the old copy, disable the loopback scheduler when you make it read-only:</p>



<pre class="wp-block-code"><code>define( 'DISABLE_WP_CRON', true );</code></pre>



<p class="wp-block-paragraph">And on the new server, set up a real system cron per site rather than leaving it to traffic, because a site with no traffic yet has no scheduler at all:</p>



<pre class="wp-block-code"><code>wp --path=/home/user/domains/example.com/public_html cron event run --due-now</code></pre>



<p class="wp-block-paragraph">Wrap that in a cron entry or a systemd timer at whatever interval the site needs, and check <code>wp cron event list</code> before assuming nothing important is queued.</p>



<p class="wp-block-paragraph">Email is the parallel problem and it fails quietly in the other direction. Your sites now send from a new IP. If SPF still authorizes only the old server&#8217;s address, and DKIM keys did not come across, and DMARC is set to anything stricter than none, your transactional mail starts landing in spam. Nothing bounces visibly. Password resets just stop arriving and you hear about it in support tickets a week later.</p>



<p class="wp-block-paragraph">Three records to check per domain. MX is the one not to touch casually: if mail lives elsewhere and you replace a zone wholesale, the mailboxes go with it.</p>



<ul class="wp-block-list">
<li><strong>MX:</strong> unchanged unless mail is genuinely moving with you.</li>
<li><strong>SPF:</strong> add the new server&#8217;s IP before cutover, remove the old one after the overlap window closes, not during.</li>
<li><strong>DKIM:</strong> keys are per-server. Either copy them or regenerate and publish the new public key.</li>
</ul>



<p class="wp-block-paragraph">The more resilient answer is to stop sending through the server at all. Routing WordPress mail through an authenticated SMTP relay means outbound deliverability stops depending on which machine the site happens to live on, which makes the next migration a non-event.</p>



<h2 class="wp-block-heading">Troubleshooting the weekend</h2>



<p class="wp-block-paragraph"><strong>Redirect loop on the new server.</strong> Usually <code>siteurl</code> and <code>home</code> disagreeing with what the vhost or reverse proxy is doing about HTTPS. Check both values with <code>wp option get siteurl</code> and <code>wp option get home</code>. If the site sits behind a proxy or CDN that terminates TLS, WordPress may see HTTP on the back end and redirect forever.</p>



<p class="wp-block-paragraph"><strong>Site loads, media does not.</strong> Either uploads did not finish syncing, ownership is wrong, or database paths still reference the old host. Compare a file listing on both sides before assuming it is a database problem.</p>



<p class="wp-block-paragraph"><strong>Widgets or theme options empty after import.</strong> This is the serialization damage described earlier. It is not recoverable by re-running the replacement. Restore the database from the dump and redo the URL change with WP-CLI.</p>



<p class="wp-block-paragraph"><strong>Certificate issuance failing repeatedly.</strong> Confirm what the challenge is actually hitting before retrying, because each failure spends part of your hourly authorization budget.</p>



<pre class="wp-block-code"><code>dig +noall +answer example.com A
dig +noall +answer @1.1.1.1 example.com A
openssl s_client -connect NEW_SERVER_IP:443 -servername example.com &lt; /dev/null 2&gt;/dev/null | openssl x509 -noout -subject -dates</code></pre>



<p class="wp-block-paragraph">The first query uses your local resolver, the second asks a public one directly, and the difference between them tells you whether you are looking at a real propagation state or your own DNS cache.</p>



<p class="wp-block-paragraph"><strong>Site fine, admin slow.</strong> Often a plugin trying to reach an external service that whitelisted the old IP. Look for API integrations, license checks and payment gateways in the manifest&#8217;s awkward-ones column.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Lowering TTL on cutover day. The old value is still cached; the change has no effect for as long as the old TTL says.</li>
<li>Cancelling the old server as soon as the sites load. Keep it running, read-only, for the whole overlap plus a margin. It is your rollback.</li>
<li>Replacing an entire DNS zone instead of editing the A record, and taking MX, SPF and verification TXT records with it.</li>
<li>Running search-replace before importing, or on the SQL file instead of the live database.</li>
<li>Testing only the homepage. Homepages are static and cached. Test a login, a form submission, an admin page and an uploaded image.</li>
<li>Leaving both schedulers live and finding out through duplicate customer emails.</li>
<li>Assuming the new server&#8217;s PHP extension set matches the old one because the version number does.</li>
<li>Treating rollback as a fleet-level decision. Per-site rollback is what keeps one bad site from stalling nineteen good ones.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a fleet move</h2>



<ul class="wp-block-list">
<li>Write the manifest first and update it as you go. It is the migration record and the handover document.</li>
<li>Take a restorable backup of each site to somewhere off both servers before you start, and test restoring one. A backup you have not restored is a hypothesis.</li>
<li>Warm sync early, delta sync at cutover, dump databases last. Batch by risk, and prove the stack on sites nobody will miss.</li>
<li>Verify every site from outside with a resolved request before its DNS moves.</li>
<li>Point uptime monitoring at the new IP before cutover, not after. UptimeRobot or Better Stack will tell you about a broken site faster than the client will.</li>
<li>Make the old copy read-only at the moment of DNS change, not at the end of the weekend.</li>
<li>Keep a rollback note per site: which records changed, what they were before, who can revert them.</li>
<li>Re-check cache configuration after the move rather than assuming it carried over, whether that is WP Rocket, WP Fastest Cache or a server-level layer.</li>
<li>Retire the old server on a scheduled date with a final archived backup, not by forgetting to renew it.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a bulk WordPress migration of 20 sites realistic in one weekend?</h3>



<p class="wp-block-paragraph">Yes, if the preparation happens in the week before. The weekend itself should be delta syncs, database moves and DNS changes. If you are still installing PHP extensions on Saturday morning, the weekend is not long enough.</p>



<h3 class="wp-block-heading">How low should I set the TTL, and how far ahead?</h3>



<p class="wp-block-paragraph">A low value like 300 seconds, set at least 24 to 48 hours before cutover so the previous TTL has fully expired everywhere. Going much lower than 300 buys little and increases query load. Raise it back after the migration settles.</p>



<h3 class="wp-block-heading">Will moving hosts hurt my search rankings?</h3>



<p class="wp-block-paragraph">Changing the server behind an unchanged domain is not itself a ranking event. What hurts is what goes wrong during it: 5xx responses while crawlers visit, failed certificates, redirect loops, a staging copy left indexable. Keep URLs identical, keep the site responding, and check Search Console afterwards.</p>



<h3 class="wp-block-heading">Can I use a migration plugin instead of the command line for this?</h3>



<p class="wp-block-paragraph">For one site, plugins are a reasonable choice and handle serialized data correctly. At twenty the arithmetic changes: twenty manual export and import cycles through a browser, with no dry run and no delta sync. The command line is repeatable and scriptable, which is the point at fleet scale.</p>



<h3 class="wp-block-heading">How long should I keep the old VPS running?</h3>



<p class="wp-block-paragraph">At least a week after the last site cuts over, in read-only mode. Some problems only appear over a full traffic cycle: weekly scheduled jobs, integrations that call in periodically, mail arriving at the old MX. Keep an archived backup after you destroy it.</p>



<h3 class="wp-block-heading">What if one site breaks after cutover?</h3>



<p class="wp-block-paragraph">Revert that site&#8217;s A record and lift the read-only restriction on the old copy. That is why the old server stays up and why rollback is per site. If you sync the database back afterwards, be careful not to overwrite writes that landed on the new server in the meantime.</p>



<h3 class="wp-block-heading">Do I need to move email at the same time?</h3>



<p class="wp-block-paragraph">No, and separating them is usually wiser. Move web first, confirm it is stable, then plan mailbox migration as its own change with its own window. Combining them means a single mistake in one DNS zone takes down both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A bulk WordPress migration is not twenty small migrations. It is one process run twenty times, and the difference is that at scale the visible failures get caught while the invisible ones do not.</p>



<p class="wp-block-paragraph">The copy is the easy part. rsync and WP-CLI are reliable, well documented and boring. What costs you is the overlap window: two servers both answering, both writing, both scheduling, for hours after you have mentally finished. Decide before the weekend what happens to writes on the old server, make that decision per site, and act on it at the moment DNS changes rather than at the end.</p>



<p class="wp-block-paragraph">Do that, batch by risk, and keep a rollback that works at the level of a single site, and twenty sites in a weekend is an ordinary piece of work rather than a gamble.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help moving a fleet of sites?</h2>



<p class="wp-block-paragraph">Migrations are a good thing to have a second pair of hands on, because the expensive mistakes are the quiet ones. This is work I do regularly:</p>



<ul class="wp-block-list">
<li>Auditing an existing fleet and producing the migration manifest, batch order and rollback plan before anything moves</li>
<li>Provisioning and hardening the destination VPS, including multiple PHP versions, control panel setup and firewall rules</li>
<li>Running the sync and database cutover with WP-CLI, including URL replacement on serialized data</li>
<li>Planning TLS issuance around ACME rate limits, including DNS-01 validation and automated renewal</li>
<li>Sorting out post-migration email: SPF, DKIM, DMARC and moving transactional mail onto an authenticated relay</li>
<li>Sitting in as the on-call engineer for the cutover window itself, or reviewing a plan you have already written</li>
</ul>



<p class="wp-block-paragraph">If you are planning a move and something in it worries you, send me the concrete thing: the site list, the current vhost config, the rsync output, the certbot error. I would rather look at the actual output than talk in general terms.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/bulk-wordpress-migration-vps/">Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</title>
		<link>https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Cron]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Decommissioning]]></category>
		<category><![CDATA[DKIM]]></category>
		<category><![CDATA[DNS Migration]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Restore Testing]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Synthetic Monitoring]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[Uptime Monitoring]]></category>
		<category><![CDATA[Website Migration]]></category>
		<category><![CDATA[WP-Cron]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=278</guid>

					<description><![CDATA[<p>Migrations rarely fail loudly. The site stays up while mail, cron, certificate renewal and the write path break silently behind it. A post-migration checklist for the first 48 hours, ordered by how long each failure can hide rather than by subsystem.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/">Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket you don&#8217;t want after a migration doesn&#8217;t say the site is down. It says nobody has received a contact form email since Friday.</p>



<p class="wp-block-paragraph">The site never went down. The uptime monitor stayed green all weekend. Analytics looked normal. And a few days of enquiries went somewhere nobody thought to look.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most migration damage. Not the 502. A 502 gets fixed inside ten minutes because everyone sees it at once, including the client. The expensive failures are the ones with no symptom on the front end: mail that authenticates on the old IP and not the new one, a cron job that never came across, a certificate that is valid today and will fail to renew long after everyone has stopped paying attention, orders written to a database that is about to be deleted.</p>



<p class="wp-block-paragraph">This post-migration checklist covers the first 48 hours after cutover. It&#8217;s ordered by when each failure actually becomes visible, not by category, because that ordering is what decides which problems you catch yourself and which ones reach you as a client ticket a fortnight later.</p>



<h2 class="wp-block-heading">Why a post-migration checklist should be ordered by detection latency</h2>



<p class="wp-block-paragraph">Most migration checklists you&#8217;ll find are grouped by subsystem: DNS, SSL, email, SEO, performance. That&#8217;s a fine way to write a document and a bad way to work a cutover, because it treats a 500 error on the homepage and a broken certificate renewal as equally urgent. They aren&#8217;t. One will be reported to you within minutes by someone who isn&#8217;t you. The other has a detection latency measured in weeks, and it lands at 3am.</p>



<p class="wp-block-paragraph">So sort your checks by how long the failure can hide. Short latency, catch it reactively. Long latency, go and look for it on purpose, while you still have the old server, the old data and a rollback path. That&#8217;s what makes 48 hours the right frame: long enough for a full daily cron cycle and for DNS to settle at any sane TTL, short enough that the old server is still sitting there with the evidence on it.</p>



<h2 class="wp-block-heading">Hour 0 to 1: prove you are actually looking at the new server</h2>



<p class="wp-block-paragraph">Almost everyone starts by loading the site in a browser and saying it looks fine. That check is worth close to nothing an hour after cutover. Your laptop may still hold the old answer, your ISP resolver definitely might, and if you added a hosts entry during testing you&#8217;re now testing the thing you already tested. Ask several resolvers directly, then bypass DNS entirely and talk to the new IP.</p>



<pre class="wp-block-code"><code># What is each public resolver answering, and how long will it keep answering it?
dig example.com A @1.1.1.1
dig example.com A @8.8.8.8
dig example.com A @9.9.9.9

# Talk to the new server directly, ignoring DNS completely
curl -sI --resolve example.com:443:203.0.113.10 https://example.com/

# Which certificate is that IP presenting for that hostname?
openssl s_client -connect 203.0.113.10:443 -servername example.com &lt;/dev/null 2&gt;/dev/null 
  | openssl x509 -noout -subject -issuer -dates</code></pre>



<p class="wp-block-paragraph">The full <code>dig</code> answer section, not <code>+short</code>, matters here because it shows the remaining TTL. That number tells you how much longer stale answers can survive, which is the single most useful figure you have in the first hour.</p>



<p class="wp-block-paragraph">The <code>--resolve</code> flag pins a hostname to an address for that one request, so you get the new server&#8217;s response with the correct SNI and Host header. It&#8217;s the closest thing to a definitive answer you&#8217;ll get while records are still in flight. If you want to see what the rest of the world sees, an exit in another country helps, and a VPN account with NordVPN or Surfshark is a cheaper way to get one than spinning up a box in a second region.</p>



<p class="wp-block-paragraph">Things to confirm in this window:</p>



<ul class="wp-block-list">
<li>The response comes from the new server, verified by something unique to it rather than by the page looking right. A temporary header or a file at a path that only exists on the new box removes all doubt. <em>Pass test: you can name the specific artefact that proves it.</em></li>

<li>The correct virtual host is selected, not a catch-all default. Sites with several domains on one server fail here quietly, serving the wrong document root over a valid certificate. <em>Pass test: every hostname you migrated returns its own content, checked individually.</em></li>

<li>The certificate chain is complete on the new box, not just the leaf. Browsers often paper over a missing intermediate using cached data; API clients and payment callbacks do not. <em>Pass test: an external SSL checker reports a complete chain from a machine that has never visited the site.</em></li>

<li>Redirect behaviour matches the old server exactly. HTTP to HTTPS, www to apex or the reverse, trailing slashes. <em>Pass test: a handful of known deep URLs return the same final status and location as before.</em></li>

<li>Any hosts file entries you added for testing are gone. Leave one in place and you&#8217;ll misread every check you run for the next two days.</li>
</ul>



<h2 class="wp-block-heading">Hour 1 to 6: the split-brain window, where you lose data instead of uptime</h2>



<p class="wp-block-paragraph">This is the part that costs real money, and it&#8217;s the part almost nobody checks, because during it both servers are healthy. Some resolvers still hand out the old address. Those visitors reach the old application, which happily accepts their form submission, their order, their uploaded file, their account signup. Then you delete that server.</p>



<p class="wp-block-paragraph">&#8220;Keep the old server running for 48 hours&#8221; is standard advice and it&#8217;s necessary, but on its own it makes the problem worse rather than better. A running old server that still accepts writes is a data-loss machine. What you want is an old server that stays reachable and stops accepting writes.</p>



<p class="wp-block-paragraph">Three options, in order of how much I&#8217;d trust them:</p>



<ol class="wp-block-list">
<li>Proxy the old server to the new one. Every request that lands on the old address gets forwarded to the new backend, so there is only ever one database. This is the only option that genuinely eliminates the split, and it&#8217;s worth the extra config on anything transactional.</li>

<li>Redirect the old server to a hostname that only resolves to the new IP. Cheap, works for GET traffic, does nothing useful for a POST that&#8217;s already in flight.</li>

<li>Put the old application into a read-only or maintenance state. Safe, but visitors on stale DNS see a broken site for the length of your TTL, so this only works if that number is genuinely small.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, go and look at what the old box actually did after the cutover.</p>



<pre class="wp-block-code"><code># On the OLD server: has anything been written since cutover?
grep '"POST ' /var/log/nginx/access.log | tail -n 50

# Apache layouts vary; on DirectAdmin the per-domain log lives here
grep '"POST ' /var/log/httpd/domains/example.com.log | tail -n 50

# Is the old box still receiving mail for the domain?
tail -n 200 /var/log/maillog</code></pre>



<p class="wp-block-paragraph">Every POST in that log after the cutover timestamp is a potential lost record. Take the list to the application: compare row counts or highest IDs on the write-heavy tables between old and new, and reconcile before you decommission anything.</p>



<p class="wp-block-paragraph">Mail deserves its own pass. If MX records changed, messages go to whichever server the sending system had cached, and land in two different mailboxes. Keep both able to accept mail through the overlap, and check the old mailbox before you close the account, not after.</p>



<p class="wp-block-paragraph">Two more things here that won&#8217;t show up in any log you&#8217;re watching:</p>



<ul class="wp-block-list">
<li><strong>Anything pinned to an IP address.</strong> Payment gateway allowlists, webhook endpoints, database firewall rules, SMTP relay authorisations, third-party API allowlists. None of these follow a redirect. A webhook posted to the old IP returns a perfectly good 200 and the new application never hears about the payment.</li>

<li><strong>Outbound connections from the new IP.</strong> Your application is now calling other people&#8217;s APIs from an address they&#8217;ve never seen. If any of those partners allowlist by IP, the failure appears as a timeout in a background job rather than an error on a page.</li>
</ul>



<h2 class="wp-block-heading">Hour 6 to 24: everything that runs on a schedule</h2>



<p class="wp-block-paragraph">Files and databases get migrated. Scheduled work frequently does not, because it lives outside the document root and outside the database dump, and because nothing on the site breaks when it&#8217;s missing.</p>



<pre class="wp-block-code"><code># Every user's crontab, not just root's
for u in $(cut -d: -f1 /etc/passwd); do echo "== $u"; crontab -l -u "$u" 2&gt;/dev/null; done

# systemd timers, where a lot of "cron" actually lives now
systemctl list-timers --all

# WordPress: what is scheduled, and is anything overdue?
wp cron event list --fields=hook,next_run_relative,recurrence
wp config get DISABLE_WP_CRON</code></pre>



<p class="wp-block-paragraph">Run that on both servers and diff the output. That comparison is the whole check, and it takes two minutes.</p>



<p class="wp-block-paragraph">The specific traps worth knowing:</p>



<ul class="wp-block-list">
<li><strong>The old server&#8217;s cron is still running.</strong> Nobody stops it, because &#8220;keep it up for 48 hours&#8221; sounded like keep everything up. Now your nightly export runs twice, your reminder emails send twice, and your billing job runs against a database nobody is reading. Disable cron on the old box at cutover, before you disable anything else.</li>

<li><strong>WP-Cron was disabled in config and the replacement never came across.</strong> If <code>DISABLE_WP_CRON</code> is true and no system cron hits <code>wp-cron.php</code>, every scheduled task stops silently. Scheduled posts sit at &#8220;Missed schedule&#8221;, backups stop, WooCommerce action queues back up. The site itself works perfectly.</li>

<li><strong>The cron URL is wrong in a way that still returns success.</strong> A job calling an <code>http://</code> URL that now redirects to HTTPS gets a 3xx, exits zero, and looks healthy in the log while the task never executes. Same story for a cron still pointing at the old hostname or an old absolute path.</li>

<li><strong>Backups.</strong> The job may have moved and still be writing to storage credentials, a mount, or a destination path that only existed on the old host. Don&#8217;t check that the backup job ran. Check that a file arrived at the destination, and that it&#8217;s the size you expect.</li>

<li><strong>Log rotation.</strong> A fresh server often has no rotation configured for application logs. Nothing goes wrong for three days, then the disk fills at 2am and every write fails at once.</li>
</ul>



<h2 class="wp-block-heading">Hour 24 to 48: the delayed detonations</h2>



<p class="wp-block-paragraph">These are the checks with the longest detection latency, which makes them the ones most worth doing deliberately.</p>



<h3 class="wp-block-heading">Certificate renewal, not the certificate</h3>



<p class="wp-block-paragraph">A copied certificate is valid for weeks. What breaks in a migration is the machinery that replaces it: the webroot path in the renewal config no longer exists, the ACME account keys weren&#8217;t copied, port 80 is closed on the new firewall, or a WAF is now intercepting the challenge path. None of that surfaces until the renewal window opens, long after everyone has moved on.</p>



<pre class="wp-block-code"><code># What does certbot think it manages, and how does it validate?
certbot certificates

# The only check that matters: will the NEXT renewal succeed?
certbot renew --dry-run

# Is anything actually going to run it?
systemctl list-timers | grep -i certbot</code></pre>



<p class="wp-block-paragraph">The dry run validates against Let&#8217;s Encrypt&#8217;s staging environment, so it exercises the real challenge path without consuming your issuance quota. Resist the urge to fix a failure with a forced renewal: Let&#8217;s Encrypt applies rate limits per registered domain and a separate limit on identical certificates, and burning through them during a migration is how a fixable config problem turns into a week of waiting.</p>



<h3 class="wp-block-heading">Mail identity on an address with no history</h3>



<p class="wp-block-paragraph">If the new server sends mail, it&#8217;s sending from an IP that receiving systems have never seen. Authentication correctness and reputation are two separate problems and they fail differently. Authentication failures bounce or land in spam immediately. Reputation problems look like intermittent delivery to one provider and normal delivery to another.</p>



<pre class="wp-block-code"><code># Reverse DNS for the new sending IP
dig +short -x 203.0.113.10

# Does SPF still authorise everything that sends for you?
dig +short TXT example.com

# DKIM selector (replace 'selector') and DMARC policy
dig +short TXT selector._domainkey.example.com
dig +short TXT _dmarc.example.com</code></pre>



<p class="wp-block-paragraph">Send a test message to an address at a major provider and read the full headers. The Authentication-Results header tells you whether SPF, DKIM and DMARC each passed, and crucially whether the passing domain aligns with the visible From address. SPF can pass for the envelope sender and still fail DMARC when the From domain differs, which is a failure mode you cannot see from the sending side.</p>



<p class="wp-block-paragraph">Leave the old server&#8217;s SPF authorisation and DKIM public key published through the overlap. Mail that was queued or delayed on the old path still needs to verify. Sources differ on how long to keep them, and the honest answer is: until you have confirmed nothing is still sending or retrying from the old host.</p>



<p class="wp-block-paragraph">If transactional mail matters to the business, the migration is a reasonable moment to stop sending it from a web server at all. Handing it to a dedicated provider such as Postmark, Mailgun or Amazon SES moves reputation off an address you&#8217;ve just changed and gives you delivery logs you can actually query.</p>



<h3 class="wp-block-heading">Search crawl, redirects and staging leftovers</h3>



<p class="wp-block-paragraph">Crawlers hit harder than usual after a move, because they&#8217;re re-checking old URLs and discovering new ones at the same time. That surge is also your best early warning: it surfaces broken redirects faster than your own testing will.</p>



<ul class="wp-block-list">
<li>Fetch <code>robots.txt</code> from the live site and read it. A staging <code>Disallow: /</code> that survives cutover is the single most damaging thing on this list.</li>

<li>View source on several page types and check for a leftover noindex tag and for canonical tags pointing at a staging hostname.</li>

<li>Watch 404s in the access log rather than in a crawler report. It&#8217;s your missed-redirects list, sorted by how much anyone cares.</li>

<li>Leave the old URLs crawlable through the redirect. Blocking them means the redirects are never seen.</li>
</ul>



<h3 class="wp-block-heading">Resource behaviour under real traffic</h3>



<p class="wp-block-paragraph">Staging never told you what the process pool does at peak. Twenty-four hours of production traffic does.</p>



<pre class="wp-block-code"><code># Status code distribution since cutover
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# Disk, and the one people forget
df -h
df -i</code></pre>



<p class="wp-block-paragraph">Compare the ratio of 5xx to total requests against the old server&#8217;s logs for the same weekday. Absolute numbers mislead here; the ratio doesn&#8217;t. Watch memory across a full daily cycle rather than at a single moment, because the interesting shape is a slow climb into a limit, not a spike.</p>



<h2 class="wp-block-heading">Before you decommission the old server</h2>



<p class="wp-block-paragraph">The old server is the only copy of some things, and none of them are in your backup.</p>



<ul class="wp-block-list">
<li>Access and error logs covering the cutover window, which is your evidence for anything that surfaces next month.</li>

<li>Mail spools, and any messages that arrived after the MX change.</li>

<li>The full crontab set, exported to a file rather than a screenshot.</li>

<li>Web server, PHP and database configuration, so you can diff a behavioural difference you notice later.</li>

<li>A final database dump taken after write traffic stopped.</li>
</ul>



<p class="wp-block-paragraph">Keep the TTL low until you&#8217;ve decided you&#8217;re staying. Raising it back to normal is the last step of the migration, not the first thing you do once the site loads. Another month on an InterServer or Contabo VPS is cheap next to reconstructing a week of orders, and the overlap is the only period in which rolling back is still a five-minute operation.</p>



<h2 class="wp-block-heading">Troubleshooting: three symptoms and what they usually mean</h2>



<p class="wp-block-paragraph"><strong>&#8220;It works for me but not for them.&#8221;</strong> Almost always a resolution split. Get the affected user to report what their resolver returns, then check whether the answer is stale DNS, a corporate resolver ignoring your TTL, or a CDN edge still holding the old origin. If you use Cloudflare or a similar proxy in front, remember the origin change and the DNS change are two separate things and only one of them propagates.</p>



<p class="wp-block-paragraph"><strong>&#8220;The site is fine but our mail goes to spam now.&#8221;</strong> Work in this order: reverse DNS for the sending IP, then SPF including the new address, then DKIM signing on the new host, then DMARC alignment with the visible From domain. Only after all four pass should you start thinking about reputation and volume, and reputation on a new IP is a matter of ramping gradually rather than something you can configure.</p>



<p class="wp-block-paragraph"><strong>&#8220;Random 500s that never happened in staging.&#8221;</strong> Look at limits before you look at code. Process pool size, memory limits, database connection caps, open file limits, and file ownership on directories the application writes to. Permission problems after an rsync or archive restore are common and produce errors that look nothing like permission problems from the front end.</p>



<h2 class="wp-block-heading">Common mistakes in the first 48 hours</h2>



<ul class="wp-block-list">
<li>Treating a green uptime check as verification. It confirms one URL returns 200. It says nothing about mail, cron, writes or checkout.</li>

<li>Leaving the old server fully operational, cron and all, instead of leaving it reachable but inert.</li>

<li>Testing while a hosts file entry is still in place.</li>

<li>Raising the TTL back to normal on day one, which throws away your fast rollback.</li>

<li>Checking that the certificate is valid instead of checking that renewal works.</li>

<li>Silencing monitoring for the maintenance window and forgetting to unsilence it.</li>

<li>Declaring the migration done before a full daily cron cycle has completed on the new server.</li>
</ul>



<h2 class="wp-block-heading">If you only do five things</h2>



<ol class="wp-block-list">
<li>Stop writes and cron on the old server, but keep it reachable.</li>

<li>Run <code>certbot renew --dry-run</code> and confirm a timer or cron will actually invoke it.</li>

<li>Send a real test email to a major provider and read the Authentication-Results header.</li>

<li>Diff the scheduled jobs on both servers, then confirm one real backup file landed at its destination.</li>

<li>Grep the old server&#8217;s access log for POST requests after the cutover, and reconcile anything you find.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How long should I keep the old server running after a migration?</h3>



<p class="wp-block-paragraph">Long enough that no resolver anywhere is still handing out the old address, plus enough margin to reconcile data and pull logs. For most sites that&#8217;s a few days. The cost of the extra billing period is trivial compared with losing your rollback path, but keep it inert: reachable, not writable, with cron disabled.</p>



<h3 class="wp-block-heading">Why is my site fine but email broken after a server migration?</h3>



<p class="wp-block-paragraph">Because web and mail are validated by different systems using different records. The web path only needs an A record and a certificate. Mail needs the MX record, reverse DNS on the sending IP, SPF listing that IP, DKIM signing configured on the new host, and DMARC alignment with your From domain. A migration can satisfy the first set completely and break every item in the second.</p>



<h3 class="wp-block-heading">What should I check first if something looks wrong right after cutover?</h3>



<p class="wp-block-paragraph">Establish which server you are talking to before you diagnose anything else. Use <code>curl --resolve</code> to pin the hostname to the new IP and compare that response with what you get through normal DNS. Half of post-cutover panic is people debugging the old server.</p>



<h3 class="wp-block-heading">Can I lose form submissions or orders during DNS propagation?</h3>



<p class="wp-block-paragraph">Yes, and it&#8217;s the most common serious loss in a migration. Visitors on cached DNS reach the old application, which writes to the old database. Nothing errors. Proxying the old server to the new backend removes the problem entirely; short of that, freeze writes during the overlap and check the old access log for POST requests before decommissioning.</p>



<h3 class="wp-block-heading">Why did my SSL certificate stop renewing weeks after the migration?</h3>



<p class="wp-block-paragraph">The certificate itself migrated fine; the renewal configuration didn&#8217;t. Common causes are a webroot path that no longer exists, an ACME account directory that wasn&#8217;t copied, port 80 blocked by the new firewall, a proxy or WAF intercepting the challenge path, or no timer scheduled to run the renewal at all. A dry run within 48 hours catches all of these.</p>



<h3 class="wp-block-heading">Do I need to tell search engines about a server migration?</h3>



<p class="wp-block-paragraph">If the URLs are unchanged and only the hosting moved, no announcement is required. Your job is to make sure crawlers can reach the new server, that nothing carries a staging noindex or a blocking robots rule, and that canonical tags point at live URLs. If the domain or URL structure changed, that&#8217;s a different migration with its own redirect and site-move process.</p>



<h3 class="wp-block-heading">What monitoring should I add specifically for the 48-hour window?</h3>



<p class="wp-block-paragraph">Add checks for the things a homepage ping cannot see: a synthetic transaction through a real form or checkout, a heartbeat from each scheduled job, certificate expiry, and a content check that asserts the right text is present rather than just a 200 status. Most uptime platforms handle the first and last; heartbeat monitoring for cron is the one people skip and the one that catches the silent failures.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A migration doesn&#8217;t fail when the site goes down. It fails when the site stays up and something behind it stops, quietly, in a way nobody is looking for. Every item on this post-migration checklist exists because the failure it catches produces no visible symptom on the front end.</p>



<p class="wp-block-paragraph">So work the 48 hours in order of detection latency. Prove which server you&#8217;re on, close the write path on the old one, diff the scheduled jobs, dry-run the certificate renewal, and read the headers on a real test email. Then, and only then, raise the TTL and let the old box go.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on a cutover?</h2>



<p class="wp-block-paragraph">Migrations are one of the few pieces of infrastructure work where the review matters more than the execution. Things I help with:</p>



<ul class="wp-block-list">
<li>Planning the cutover so the write path is closed on the old server rather than left open, including proxy-forward setups for transactional sites.</li>

<li>Running the 48-hour verification pass on a migration someone else performed, and telling you plainly what&#8217;s still broken.</li>

<li>Repairing certificate renewal, ACME challenge paths and reload hooks that stopped working after the move.</li>

<li>Fixing mail on a new IP: reverse DNS, SPF, DKIM signing and DMARC alignment, or moving transactional mail off the web server entirely.</li>

<li>Rebuilding scheduled jobs, backup destinations and log rotation on the new host, with heartbeat monitoring so silence becomes an alert.</li>

<li>Reconciling data written to the old server during the split window, before anything gets deleted.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re mid-migration or just came out of one, send me something concrete: the old server&#8217;s access log around the cutover, your crontab, or the headers from a test email. That&#8217;s usually enough to tell you what&#8217;s wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/post-migration-checklist-48-hours/">Post-Migration Checklist: What Breaks Quietly in the First 48 Hours</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</title>
		<link>https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Automatic HTTPS]]></category>
		<category><![CDATA[Caddy]]></category>
		<category><![CDATA[Caddyfile]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[mod_php]]></category>
		<category><![CDATA[MPM]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reverse Proxy]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[Web Server]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=195</guid>

					<description><![CDATA[<p>Benchmarks pick the wrong winner. Here is how Nginx, Apache and Caddy actually differ in production: how each one fails under load, who owns certificate renewal, and where configuration lives when a developer needs to change it. Includes a decision procedure and the mistakes I see most.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A message landed in a channel I was in: &#8220;We&#8217;re standardising on Nginx.&#8221; No traffic profile. No mention of what the current setup was actually failing at. No list of the applications that would have to move. The decision got made because everybody knows Nginx is fast.</p>



<p class="wp-block-paragraph">That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.</p>



<p class="wp-block-paragraph">So this post compares <strong>Nginx vs Apache vs Caddy</strong> on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.</p>



<h2 class="wp-block-heading">Why the benchmark comparison picks the wrong winner</h2>



<p class="wp-block-paragraph">Almost every throughput comparison you&#8217;ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you&#8217;re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.</p>



<p class="wp-block-paragraph">The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn&#8217;t most projects.</p>



<p class="wp-block-paragraph">For everyone else, the useful question is not &#8220;which is fastest&#8221; but &#8220;which chores am I choosing to keep.&#8221; That&#8217;s what the rest of this is about.</p>



<h2 class="wp-block-heading">Failure family one: the process model decides how you fall over</h2>



<p class="wp-block-paragraph">This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.</p>



<h3 class="wp-block-heading">Apache: a hard worker ceiling, and a queue behind it</h3>



<p class="wp-block-paragraph">Apache&#8217;s behaviour depends entirely on which Multi-Processing Module is loaded. With <code>prefork</code>, you get one process per connection, capped by <code>MaxRequestWorkers</code>. When you hit that cap, new connections don&#8217;t get rejected. They sit in the kernel&#8217;s listen backlog waiting for a worker to free up.</p>



<p class="wp-block-paragraph">The symptom is the worst kind: the site &#8220;hangs.&#8221; Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.</p>



<p class="wp-block-paragraph">Check which MPM you&#8217;re actually running before you theorise about anything:</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"</code></pre>



<p class="wp-block-paragraph">The reason prefork survives is <code>mod_php</code>. It isn&#8217;t safe with Apache&#8217;s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over <code>mod_proxy_fcgi</code> instead, which frees you to run <code>mpm_event</code> and gets you HTTP/2 as a side effect, since <code>mod_http2</code> disables itself under prefork.</p>



<p class="wp-block-paragraph">If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.</p>



<h3 class="wp-block-heading">Nginx: cheap connections, different limits</h3>



<p class="wp-block-paragraph">Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.</p>



<p class="wp-block-paragraph">Its ceiling is <code>worker_processes</code> multiplied by <code>worker_connections</code>, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That&#8217;s a genuine operational advantage: the failure announces itself.</p>



<p class="wp-block-paragraph">The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.</p>



<h3 class="wp-block-heading">Caddy: goroutines, and a softer edge</h3>



<p class="wp-block-paragraph">Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don&#8217;t tune a worker count at all.</p>



<p class="wp-block-paragraph">The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean &#8220;workers exhausted&#8221; line, you get memory growth and garbage collection pressure. There&#8217;s no knob you turned wrong, which is pleasant right up until you need to explain what happened.</p>



<h2 class="wp-block-heading">Failure family two: who owns the certificate lifecycle</h2>



<p class="wp-block-paragraph">This is the one that actually causes outages, and it is the sharpest genuine difference between the three.</p>



<p class="wp-block-paragraph">With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.</p>



<p class="wp-block-paragraph">The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:</p>



<pre class="wp-block-code"><code># Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"</code></pre>



<p class="wp-block-paragraph">The distinction that matters: <code>--deploy-hook</code> runs only when a certificate was actually renewed, so you&#8217;re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.</p>



<p class="wp-block-paragraph">Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.</p>



<p class="wp-block-paragraph">It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy&#8217;s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an <code>ask</code> endpoint you host:</p>



<pre class="wp-block-code"><code>{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}</code></pre>



<p class="wp-block-paragraph">Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older <code>interval</code> and <code>burst</code> rate-limiting options are no longer recommended and should be removed if you still have them in a config.</p>



<p class="wp-block-paragraph">One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.</p>



<h2 class="wp-block-heading">Failure family three: where the configuration lives</h2>



<p class="wp-block-paragraph">This is the axis people underweight, and it&#8217;s usually the one that actually decides the answer.</p>



<p class="wp-block-paragraph">Apache supports <code>.htaccess</code>: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.</p>



<p class="wp-block-paragraph">The costs are real and they&#8217;re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for <code>.htaccess</code> files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.</p>



<p class="wp-block-paragraph">Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with <code>try_files</code> rather than per-directory rule files. If you own the box, that&#8217;s a feature: nothing changes routing behind your back and the running configuration is exactly what&#8217;s in version control. If you host other people&#8217;s applications, it&#8217;s a support burden, because every customer rewrite rule becomes a ticket for you.</p>



<p class="wp-block-paragraph">Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Nginx: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you&#8217;ll meet has a known-good Nginx config, and everyone you hire has already read one.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the DNS behaviour catches people constantly. A hostname written directly into <code>proxy_pass</code>, or into an <code>upstream</code> block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says &#8220;stale DNS&#8221;; you just get connection timeouts to an address that looks plausible.</p>



<p class="wp-block-paragraph">Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that&#8217;s what moves the lookup from config parse time to request time:</p>



<pre class="wp-block-code"><code>location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}</code></pre>



<p class="wp-block-paragraph">Note the <code>$request_uri</code> on the end. Once a variable appears in <code>proxy_pass</code>, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream&#8217;s root. Also worth knowing: an <code>upstream</code> block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.</p>



<p class="wp-block-paragraph">The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But &#8220;who decides what ships&#8221; is a fair question to ask about any dependency, and this one now has a public answer.</p>



<h2 class="wp-block-heading">Apache: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> memory per connection, if you&#8217;re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that&#8217;s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.</p>



<p class="wp-block-paragraph">I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and <code>mod_php</code> to <code>mpm_event</code> and PHP-FPM first, then measure, and quite often stop there.</p>



<h2 class="wp-block-heading">Caddy: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.</p>



<pre class="wp-block-code"><code>app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}</code></pre>



<p class="wp-block-paragraph">That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with <code>xcaddy</code> rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there&#8217;s a subtler cost: automation you don&#8217;t understand is still automation you&#8217;re responsible for. Being paged about ACME behaviour you&#8217;ve never had to learn is a worse afternoon than being paged about Certbot.</p>



<p class="wp-block-paragraph">On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the <code>dynamic a</code> upstream module with a <code>refresh</code> interval is the built-in answer, which is a cleaner story than Nginx&#8217;s variable trick.</p>



<h2 class="wp-block-heading">A decision procedure for Nginx vs Apache vs Caddy</h2>



<p class="wp-block-paragraph">Work down this list and stop at the first one that matches. It&#8217;ll be right more often than a benchmark.</p>



<ol class="wp-block-list">
<li><strong>Do applications you don&#8217;t control need to change server behaviour?</strong> If yes, Apache. Nothing else has <code>.htaccess</code>, and reimplementing it in tickets is not a plan.</li>

<li><strong>Is TLS renewal your top operational risk?</strong> Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.</li>

<li><strong>Are you terminating a very high connection rate, or serving heavy static content?</strong> Nginx. This is the workload it was designed around.</li>

<li><strong>Does the team already run one of these well?</strong> Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.</li>

<li><strong>Is there a working Apache install and no specific complaint?</strong> Fix the MPM and PHP handler, then reassess. Most &#8220;we need Nginx&#8221; tickets are actually &#8220;we need to stop using prefork.&#8221;</li>

<li><strong>None of the above?</strong> Nginx for anything that will be handed to another engineer. Caddy for anything you&#8217;ll run yourself.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Renewing certificates without reloading.</strong> The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.</li>

<li><strong>Migrating for benchmark numbers.</strong> You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.</li>

<li><strong>Leaving Apache on prefork after moving to PHP-FPM.</strong> The migration is only half done, and you&#8217;re still paying prefork&#8217;s memory cost while also giving up HTTP/2.</li>

<li><strong>Hostnames baked into <code>proxy_pass</code> with dynamic upstreams.</strong> Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.</li>

<li><strong>Enabling Caddy&#8217;s on-demand TLS without an <code>ask</code> endpoint.</strong> You&#8217;ve handed strangers the ability to make you request certificates.</li>

<li><strong>Exposing Caddy&#8217;s admin API beyond loopback.</strong> Unauthenticated full config replacement is exactly as bad as it sounds.</li>

<li><strong>Monitoring the certificate file instead of the handshake.</strong> The file is almost never the thing that&#8217;s wrong. Check what the server actually presents, from outside the box.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide, in practice</h2>



<ul class="wp-block-list">
<li>Pick based on who has to change the config six months from now, not on requests per second today.</li>

<li>Whichever you choose, dump the effective configuration rather than reading files: <code>nginx -T</code>, <code>apachectl -S</code>, <code>caddy adapt --config /etc/caddy/Caddyfile</code>. Include files hide surprises.</li>

<li>Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.</li>

<li>Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.</li>

<li>Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.</li>

<li>Validate before reloading, every time. <code>nginx -t</code>, <code>apachectl configtest</code> and <code>caddy validate</code> all exist so you don&#8217;t find out from users.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Caddy production ready?</h3>



<p class="wp-block-paragraph">Yes. It&#8217;s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.</p>



<h3 class="wp-block-heading">Can I run Nginx and Apache together?</h3>



<p class="wp-block-paragraph">Yes, and it&#8217;s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their <code>.htaccess</code> behaviour. You get Nginx&#8217;s connection handling and Apache&#8217;s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.</p>



<h3 class="wp-block-heading">Which is best for WordPress?</h3>



<p class="wp-block-paragraph">Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into <code>try_files</code> and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.</p>



<h3 class="wp-block-heading">Does Nginx really re-resolve DNS for upstreams?</h3>



<p class="wp-block-paragraph">Not by default. A literal hostname in <code>proxy_pass</code> or an <code>upstream</code> block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a <code>resolver</code> directive, and remembering that this changes how the request URI is passed.</p>



<h3 class="wp-block-heading">Should I switch from mod_php to PHP-FPM?</h3>



<p class="wp-block-paragraph">In almost all cases yes. It&#8217;s the change that unlocks Apache&#8217;s threaded MPMs and HTTP/2, and it separates PHP&#8217;s lifecycle from the web server&#8217;s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or <code>.htaccess</code>, since those move to a <code>.user.ini</code> file with different syntax.</p>



<h3 class="wp-block-heading">Does the freenginx fork mean I should stop using Nginx?</h3>



<p class="wp-block-paragraph">No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn&#8217;t put a client on a fork without a specific reason.</p>



<h3 class="wp-block-heading">Which uses the least memory?</h3>



<p class="wp-block-paragraph">Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on <code>mpm_event</code>, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Nginx vs Apache vs Caddy</strong> question is not a performance question. All three are fast enough for what you&#8217;re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.</p>



<p class="wp-block-paragraph">Pick the set of chores your team can actually carry. Then go fix the thing that&#8217;s genuinely slow, which is almost certainly not the web server.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your web server setup?</h2>



<p class="wp-block-paragraph">Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does</li>

<li>Migrating Apache from prefork and <code>mod_php</code> to <code>mpm_event</code> with PHP-FPM, without breaking existing <code>.htaccess</code> behaviour</li>

<li>Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches</li>

<li>Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring</li>

<li>Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted</li>

<li>Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me your config, an error log excerpt, or the output of <code>nginx -T</code> and tell me what you&#8217;re seeing. I&#8217;d rather look at the real thing than guess from a description.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</title>
		<link>https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:23 +0000</pubDate>
				<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[ACME]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Certificate Chain]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[DNS]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OpenSSL]]></category>
		<category><![CDATA[SSL Certificate]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=64</guid>

					<description><![CDATA[<p>Your browser shows a padlock and your partner's API client throws "unable to get local issuer certificate". Both are correct. A practical guide to the four things a TLS client actually checks, and to renewal automation that reaches the running server instead of just the disk.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/">It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A partner emails to say your API has been rejecting their integration for two days. Their client throws <code>unable to get local issuer certificate</code>. You open the URL in Chrome, see a padlock, and reply that it works fine on your end. Then someone runs <code>curl</code> from a clean container and it fails there too.</p>



<p class="wp-block-paragraph">Your server is only sending the leaf certificate. The intermediate that links it to a trusted root is missing. Your browser papers over that: it either has the intermediate cached from another site, or it quietly fetches it using the AIA extension in the certificate. <code>curl</code>, Java, Go, Python and every other non-browser client do no such thing. They see a certificate signed by something they do not trust and stop.</p>



<p class="wp-block-paragraph">This is the most common of all <strong>TLS certificate errors</strong> and it is close to invisible from a desktop. The padlock in your browser is not a test. It is a client with a large safety net doing you a favour.</p>



<p class="wp-block-paragraph">This post covers the four things a client actually checks and how each one fails: the chain, the names, the clock, and the trust store. Then the automation half, which has its own quiet failure: a renewal that succeeds and never reaches the running server. Plus the two things about TLS that have genuinely changed recently and will break config you copied from an older guide.</p>



<h2 class="wp-block-heading">Look at what the server is actually sending</h2>



<p class="wp-block-paragraph">Before theorising, get the facts. One command tells you almost everything.</p>



<pre class="wp-block-code"><code># -servername sends SNI. Without it you get whatever the default
# vhost serves, which is a different certificate on most servers
# and the source of a lot of confusing results.
# -showcerts prints every certificate the server sent, in order.
openssl s_client -connect example.com:443 -servername example.com -showcerts &lt;/dev/null</code></pre>



<p class="wp-block-paragraph">Read the top of that output. There is a <code>Certificate chain</code> block listing what arrived, numbered from zero, with <code>s:</code> for subject and <code>i:</code> for issuer. Entry 0 is your certificate. Entry 1 should be the intermediate that issued it, and its subject should exactly match entry 0&#8217;s issuer.</p>



<p class="wp-block-paragraph">If the chain block has only entry 0, that is your bug. Right at the bottom, <code>Verify return code</code> gives the verdict: <code>0 (ok)</code> is what you want, <code>21 (unable to verify the first certificate)</code> means the chain is incomplete, and <code>20 (unable to get local issuer certificate)</code> means the issuer is not in the local trust store.</p>



<p class="wp-block-paragraph">Run it from a machine that is not yours. A clean container is ideal, because your laptop has accumulated intermediates and corporate roots that a customer&#8217;s server has not.</p>



<h2 class="wp-block-heading">Chain problems</h2>



<p class="wp-block-paragraph">Almost all of these come down to one file choice. ACME clients write out several files and two of them look interchangeable:</p>



<ul class="wp-block-list">
<li><strong><code>cert.pem</code></strong> is the leaf alone. Using this is the mistake.</li>
<li><strong><code>chain.pem</code></strong> is the intermediates alone.</li>
<li><strong><code>fullchain.pem</code></strong> is leaf plus intermediates, in the right order. This is the one your server should point at.</li>
</ul>



<pre class="wp-block-code"><code># nginx
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;</code></pre>



<pre class="wp-block-code"><code># Apache. On current versions the chain goes in SSLCertificateFile.
# SSLCertificateChainFile is deprecated; guides that still use it
# are old enough that other advice in them is probably stale too.
SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem</code></pre>



<p class="wp-block-paragraph">Order matters as well as contents. The leaf must come first, then each intermediate that signs the one above it. A chain assembled in the wrong order will be rejected by strict clients even though every certificate in it is correct. You can check a chain offline before deploying it:</p>



<pre class="wp-block-code"><code># Does this leaf chain to a trusted root using these intermediates?
# Exits non-zero and explains itself if not.
openssl verify -untrusted chain.pem cert.pem</code></pre>



<p class="wp-block-paragraph">Do not include the root certificate in the file you serve. The client either has it or does not, and sending it wastes bytes on every handshake without changing any outcome.</p>



<h2 class="wp-block-heading">Name problems</h2>



<p class="wp-block-paragraph">The second family. A certificate can be perfectly valid and still be the wrong certificate for the name being requested.</p>



<pre class="wp-block-code"><code># The hostname the client asked for must appear in here.
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"</code></pre>



<p class="wp-block-paragraph">Three things people get wrong here:</p>



<ul class="wp-block-list">
<li><strong>The Common Name does not count.</strong> Modern clients ignore the CN field entirely for hostname matching. If the name is only in the CN and not in the SAN list, it will fail, and the error message will not explain that clearly.</li>
<li><strong>Wildcards are exactly one label deep.</strong> <code>*.example.com</code> covers <code>www.example.com</code> and <code>api.example.com</code>. It does not cover <code>example.com</code> itself, and it does not cover <code>a.b.example.com</code>. The bare domain has to be listed separately, and it usually is not.</li>
<li><strong>SNI decides which certificate you get.</strong> One IP serving many sites picks the certificate based on the name the client sends during the handshake. A client that does not send SNI, or an <code>openssl</code> command without <code>-servername</code>, gets the default vhost&#8217;s certificate and produces a name mismatch that has nothing to do with your actual configuration.</li>
</ul>



<h2 class="wp-block-heading">Time and trust problems</h2>



<pre class="wp-block-code"><code># Local file
openssl x509 -in cert.pem -noout -dates

# Live endpoint, subject and dates in one go
openssl s_client -connect example.com:443 -servername example.com &lt;/dev/null 2&gt;/dev/null 
  | openssl x509 -noout -dates -subject</code></pre>



<p class="wp-block-paragraph">Expired is obvious. <code>notBefore</code> in the future is the interesting one, because it almost never means the certificate is wrong. It means the client&#8217;s clock is wrong. Containers with no NTP, a VM restored from a snapshot, an embedded device with a dead battery: all of them produce &#8220;certificate not yet valid&#8221; against a perfectly good certificate. Check the clock on the failing machine before you touch anything on the server.</p>



<p class="wp-block-paragraph">Trust store problems look similar but come from a different place. The root that signed your chain has to be in the trust store of the machine making the request, and old systems have old trust stores. A device that has not received an OS update in years may simply not know about a certificate authority that appeared since. There is no server-side fix for that beyond choosing a CA with broader legacy coverage, which is one of the few genuine arguments for a paid certificate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The renewal that succeeded and changed nothing</h2>



<p class="wp-block-paragraph">Here is the second invisible failure, and it catches experienced people. Your renewal timer runs, the ACME client obtains a new certificate, writes it to disk, logs success. The site goes down anyway.</p>



<p class="wp-block-paragraph">Nginx read the certificate into memory when it started. It does not watch the file. Neither does Apache, HAProxy, Postfix, or almost anything else. Until the service is reloaded it keeps serving the certificate it loaded at boot, which is the one that just expired. Everything in your logs says the renewal worked, because it did. The delivery step was never wired up.</p>



<p class="wp-block-paragraph">Certbot handles this with deploy hooks, which run only when a certificate was actually renewed. Put a script in the hooks directory and it applies to every certificate on the box:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-services.sh
# Remember: chmod +x, or it silently never runs.
set -euo pipefail

# Validate the config BEFORE reloading. A reload with a broken
# config can take the site down at the exact moment you were
# trying to keep it up.
nginx -t
systemctl reload nginx

# Anything else that holds a copy of the certificate.
# systemctl reload postfix</code></pre>



<p class="wp-block-paragraph">Then prove the whole path works without waiting sixty days to find out:</p>



<pre class="wp-block-code"><code># Exercises the full renewal against the staging environment.
# Installs nothing, counts against no rate limits, and will
# fail loudly if your challenge setup is broken.
certbot renew --dry-run</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>--dry-run</code> does not run deploy hooks by default, precisely because nothing was deployed. So it validates the issuance path but not the reload. Test that separately by touching the service and confirming the served certificate changes.</p>



<h2 class="wp-block-heading">Expiry automation, and why it is no longer optional</h2>



<p class="wp-block-paragraph">This part has a date attached, so it is worth stating plainly rather than dancing around it.</p>



<p class="wp-block-paragraph">The CA/Browser Forum approved a phased reduction of maximum certificate lifetimes. The cap dropped from 398 days to 200 days on 15 March 2026, falls to 100 days on 15 March 2027, and reaches 47 days on 15 March 2029. Domain validation reuse periods shrink alongside it.</p>



<p class="wp-block-paragraph">The practical consequence: any process that involves a person remembering to renew something once a year is already broken, and by the third phase you are renewing every six weeks. Whatever you were getting away with manually, stop. ACME with a timer is the baseline. Certbot is the default choice, acme.sh is lighter and easier to script around, and Caddy obtains and renews certificates with no configuration at all, which is a genuinely good reason to consider it for a simple reverse proxy.</p>



<h3 class="wp-block-heading">Choosing a challenge type</h3>



<ul class="wp-block-list">
<li><strong>HTTP-01</strong> proves control by serving a file over port 80. Simple, and it breaks whenever port 80 is closed, redirected wholesale to HTTPS in a way that mangles the challenge path, or sitting behind a proxy that does not forward it.</li>
<li><strong>DNS-01</strong> proves control by writing a TXT record. More setup, needs an API token for your DNS provider, and it is the only option for wildcard certificates. It also works for hosts with no inbound internet access at all, which is why I reach for it on anything internal.</li>
</ul>



<p class="wp-block-paragraph">The trade-off with DNS-01 is a credential: your renewal process now holds a token that can edit your DNS zone. Scope it to the narrowest permission your provider offers, and if they support delegating just the <code>_acme-challenge</code> records to a separate zone, do that instead.</p>



<h3 class="wp-block-heading">Monitor expiry externally</h3>



<p class="wp-block-paragraph">Do not rely on the renewal system reporting its own health. If it is broken, it is also probably not telling you. Let&#8217;s Encrypt has discontinued its expiration reminder emails, so if that was quietly acting as your safety net, it is not one any more.</p>



<p class="wp-block-paragraph">Check from outside, against the live endpoint, the way a user would:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

HOST="example.com"
DAYS=21

# -checkend takes a number of seconds and exits non-zero if the
# certificate expires within that window. No parsing dates by hand.
if ! openssl s_client -connect "$HOST:443" -servername "$HOST" &lt;/dev/null 2&gt;/dev/null 
     | openssl x509 -noout -checkend $(( DAYS * 86400 )); then
  echo "TLS certificate for $HOST expires within $DAYS days" &gt;&amp;2
  exit 1
fi</code></pre>



<p class="wp-block-paragraph">If you already run Prometheus, the blackbox exporter gives you <code>probe_ssl_earliest_cert_expiry</code> for free, which is a timestamp you can alert on and graph in Grafana across every endpoint you own. That is the better answer at more than a handful of hosts, because it also catches the certificates nobody remembered existed.</p>



<h2 class="wp-block-heading">Two pieces of config that are now wrong</h2>



<p class="wp-block-paragraph"><strong>OCSP stapling with Let&#8217;s Encrypt.</strong> Let&#8217;s Encrypt removed the OCSP responder URL from newly issued certificates in May 2025 and shut down its responders in August 2025. If your config still has this, it does nothing except print a warning on every reload:</p>



<pre class="wp-block-code"><code># nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate
# Comment these out if your certificates come from Let's Encrypt.
# ssl_stapling on;
# ssl_stapling_verify on;</code></pre>



<p class="wp-block-paragraph">Nothing breaks by leaving them, but a warning that appears on every reload is a warning people learn to ignore, and then they ignore the next one too.</p>



<p class="wp-block-paragraph"><strong>Hand-tuned cipher suite lists.</strong> If you pasted a <code>ssl_ciphers</code> line from a guide more than a couple of years old, re-check it against the current Mozilla SSL Configuration Generator. Lists that were considered hardened at the time now sometimes exclude things you want or include things you do not, and the defaults have improved considerably.</p>



<h2 class="wp-block-heading">When there is a CDN in front</h2>



<p class="wp-block-paragraph">Behind Cloudflare or a similar edge, there are two separate TLS connections: browser to edge, and edge to your origin. They use different certificates and fail independently.</p>



<p class="wp-block-paragraph">This creates a specific blind spot. Your origin certificate can be expired, self-signed or serving the wrong name, and visitors will see nothing wrong, because the edge presents its own valid certificate and, in the more permissive encryption modes, does not verify the origin. It works until you switch to strict origin validation, or until the edge is bypassed, and then everything fails at once for reasons that look unrelated to anything you changed.</p>



<p class="wp-block-paragraph">Test the origin directly by name, bypassing DNS. If your origin is a VPS with a provider like InterServer and the public DNS points at the CDN, this is the only way to see what the origin is really serving:</p>



<pre class="wp-block-code"><code># --resolve forces the hostname to a specific IP without touching
# /etc/hosts, so SNI and Host header stay correct.
curl -v --resolve example.com:443:203.0.113.10 https://example.com/</code></pre>



<h2 class="wp-block-heading">Troubleshooting by error message</h2>



<h3 class="wp-block-heading">unable to get local issuer certificate</h3>



<p class="wp-block-paragraph">Missing intermediate on the server, or a trust store on the client that lacks the root. Check the server first with <code>-showcerts</code>; if the chain is complete, the problem is on the client side.</p>



<h3 class="wp-block-heading">certificate has expired, but I just renewed it</h3>



<p class="wp-block-paragraph">The service was never reloaded. Compare the dates on disk with the dates being served. If they differ, that is your answer, and the fix is a deploy hook rather than another manual renewal.</p>



<h3 class="wp-block-heading">Hostname mismatch on a name that is definitely in the certificate</h3>



<p class="wp-block-paragraph">Either SNI is not reaching the right vhost, or the name is in the Common Name rather than the SAN list, or you are hitting a wildcard at the wrong depth. Check the SAN list and re-run <code>s_client</code> with an explicit <code>-servername</code>.</p>



<h3 class="wp-block-heading">Works over IPv4, fails over IPv6 (or the reverse)</h3>



<p class="wp-block-paragraph">Two different machines are answering, and only one of them was updated. Force the family with <code>curl -4</code> and <code>curl -6</code> and compare. Load balancers with one member out of sync produce the same intermittent pattern.</p>



<h3 class="wp-block-heading">Renewal fails with a challenge error</h3>



<p class="wp-block-paragraph">For HTTP-01, confirm that <code>http://example.com/.well-known/acme-challenge/test</code> is reachable from outside without being redirected or rewritten. A blanket redirect to HTTPS is the usual culprit. For DNS-01, check that the TXT record actually appears with <code>dig</code>, and give slow-propagating providers more time.</p>



<h3 class="wp-block-heading">Everything looks right and it still fails</h3>



<p class="wp-block-paragraph">Run an external scan. SSL Labs or a local run of <code>testssl.sh</code> will check things you did not think to, including protocol support and chain issues from the perspective of clients you do not have installed.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing the server at <code>cert.pem</code> instead of <code>fullchain.pem</code>.</li>
<li>Treating a padlock in your own browser as proof the certificate is correctly deployed.</li>
<li>Testing with <code>openssl s_client</code> without <code>-servername</code> and debugging the wrong certificate.</li>
<li>Assuming <code>*.example.com</code> covers the bare domain.</li>
<li>Putting the hostname only in the Common Name.</li>
<li>Renewing without reloading the service.</li>
<li>Forgetting <code>chmod +x</code> on a deploy hook, so it silently never runs.</li>
<li>Relying on the renewal system to tell you when it is broken.</li>
<li>Including the root certificate in the served chain.</li>
<li>Leaving OCSP stapling enabled with certificates that no longer support it.</li>
<li>Ignoring the origin certificate because the CDN hides its state.</li>
<li>Debugging a &#8220;not yet valid&#8221; certificate on the server when the client&#8217;s clock is wrong.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Serve the full chain, leaf first, without the root.</li>
<li>Verify from a clean machine or container, never from the laptop you browse on.</li>
<li>Put every name you serve in the SAN list, including the bare domain alongside any wildcard.</li>
<li>Automate renewal with ACME and a timer, and run <code>certbot renew --dry-run</code> after any change to the setup.</li>
<li>Attach a deploy hook that validates the config before reloading, and make it executable.</li>
<li>Monitor expiry externally against the live endpoint, and alert with weeks of headroom.</li>
<li>Use DNS-01 for wildcards and internal hosts, with a narrowly scoped API token.</li>
<li>Keep NTP running everywhere, including inside containers.</li>
<li>Test the origin directly when there is a CDN in front, not just the public hostname.</li>
<li>Inventory your certificates, because the one that takes you down is the one nobody remembered.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Why does my certificate work in Chrome but fail in curl?</h3>



<p class="wp-block-paragraph">Your server is not sending the intermediate certificate. Browsers work around this by caching intermediates from other sites and by fetching them from the URL in the certificate&#8217;s AIA extension. Command-line tools and application HTTP libraries do neither. Switch the server to <code>fullchain.pem</code> and reload.</p>



<h3 class="wp-block-heading">What is the difference between cert.pem, chain.pem and fullchain.pem?</h3>



<p class="wp-block-paragraph">Leaf only, intermediates only, and both concatenated in the correct order. Web servers want <code>fullchain.pem</code>. The private key stays in <code>privkey.pem</code> and never goes in any of them.</p>



<h3 class="wp-block-heading">Does a wildcard certificate cover the root domain?</h3>



<p class="wp-block-paragraph">No. <code>*.example.com</code> matches exactly one label, so it covers <code>www.example.com</code> but not <code>example.com</code> and not <code>a.b.example.com</code>. Request the bare domain as an additional SAN, which most ACME clients will do if you pass both names.</p>



<h3 class="wp-block-heading">How often should certificates renew now?</h3>



<p class="wp-block-paragraph">Whatever your CA issues, renew at roughly two-thirds of the lifetime so a failed attempt has room to retry before anything expires. Since maximum lifetimes are on a published downward schedule, build for automated renewal at any interval rather than tuning a number you will have to change again.</p>



<h3 class="wp-block-heading">Should I still enable OCSP stapling?</h3>



<p class="wp-block-paragraph">Not with Let&#8217;s Encrypt certificates, which no longer carry a responder URL. If your CA still supports it, stapling remains worthwhile. Check whether your certificate actually contains an OCSP URL before enabling the directives, rather than assuming.</p>



<h3 class="wp-block-heading">Is a paid certificate more secure than a free one?</h3>



<p class="wp-block-paragraph">Cryptographically, no. The encryption is identical. What money buys is broader compatibility with old trust stores, longer validation reuse, warranties, and human support when something goes wrong. For a normal public website, a free automated certificate is the right default.</p>



<h3 class="wp-block-heading">How do I get a certificate for something with no public DNS or open ports?</h3>



<p class="wp-block-paragraph">DNS-01, on a public domain you control, pointed at whatever internal address you like. The CA only needs to see the TXT record; it never needs to reach the host. This is how internal services get publicly trusted certificates without being exposed.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Most TLS certificate errors are not about cryptography. They are about delivery: whether the server sends everything the client needs, whether the name it sends matches the name that was asked for, and whether the certificate sitting in memory is the same one sitting on disk.</p>



<p class="wp-block-paragraph">So stop checking with your browser. Check with <code>openssl s_client</code> from a machine that owes you no favours, and make sure something outside your server is watching the expiry date. As lifetimes keep shrinking, the gap between a renewal process that works and one that merely appears to work gets narrower every year.</p>



<h2 class="wp-block-heading">Certificates causing you trouble?</h2>



<p class="wp-block-paragraph">Certificate problems tend to arrive at bad moments and turn out to be one wrong file path. Things I help with:</p>



<ul class="wp-block-list">
<li>Diagnosing a certificate that works for some clients and not others, and telling you exactly which link in the chain is missing.</li>
<li>Setting up ACME renewal with deploy hooks across nginx, Apache, HAProxy, mail servers and anything else holding a copy.</li>
<li>Migrating to DNS-01 for wildcards and internal hosts, with properly scoped API credentials.</li>
<li>Certificate inventory and expiry monitoring in Prometheus and Grafana, so nothing expires without warning.</li>
<li>Preparing a certificate estate for shorter lifetimes, replacing anything that still depends on a person remembering.</li>
<li>Origin and edge TLS configuration behind a CDN, including strict origin validation.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>openssl s_client -connect yourhost:443 -servername yourhost -showcerts</code> and the error your failing client reports, and I will tell you what is wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/">It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</title>
		<link>https://john-nessime.com/blog/system-administration/tls-certificate-errors/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 30 Jul 2026 15:02:33 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OpenSSL]]></category>
		<category><![CDATA[SSL Certificate]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=31</guid>

					<description><![CDATA[<p>Almost every TLS certificate error traces back to one of three causes: a broken chain, a hostname the certificate does not cover, or an expiry nobody was watching. Here is how to tell them apart in under a minute, and how to stop the third one from ever happening again.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/tls-certificate-errors/">TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The site loads fine in your browser. Then a mobile app starts throwing handshake failures, a webhook from a payment provider stops arriving, and someone on the team runs <code>curl</code> and gets &#8220;SSL certificate problem: unable to get local issuer certificate&#8221;. Nothing changed on the server. Except something did, and your browser has been quietly covering for it.</p>



<p class="wp-block-paragraph">Nearly every TLS certificate error I have had to chase down comes back to one of three things: the chain your server sends is incomplete, the certificate does not actually cover the hostname being requested, or it expired and no alert fired. This article walks through all three — how to identify which one you are looking at, the exact commands to prove it, and how to set up renewal automation so the third category stops being a category at all.</p>



<h2 class="wp-block-heading">Why TLS certificate errors are so confusing to debug</h2>



<p class="wp-block-paragraph">Two things conspire to make this harder than it should be.</p>



<p class="wp-block-paragraph">The first is that browsers are forgiving in ways other clients are not. When a server sends an incomplete certificate chain, most modern browsers will quietly fetch the missing intermediate using the Authority Information Access extension embedded in the certificate. They fix your misconfiguration on the fly and show you a padlock. Meanwhile <code>curl</code>, Python&#8217;s <code>requests</code>, Java clients, Go services, and older Android devices do no such thing. They fail. This is why &#8220;it works in Chrome&#8221; is worth almost nothing as a diagnostic signal.</p>



<p class="wp-block-paragraph">The second is that error messages describe the symptom rather than the cause. &#8220;Unable to get local issuer certificate&#8221; sounds like a client problem. It usually is not — it usually means your server is not sending the full chain. Learning to translate the message into the actual failure is most of the work.</p>



<h2 class="wp-block-heading">One command that tells you almost everything</h2>



<p class="wp-block-paragraph">Before touching config files, look at what the server is actually sending on the wire:</p>



<pre class="wp-block-code"><code>openssl s_client -connect example.com:443 -servername example.com -showcerts &lt;/dev/null</code></pre>



<p class="wp-block-paragraph">The <code>-servername</code> flag matters more than people expect. It sets SNI, and without it you may be handed the server&#8217;s default certificate rather than the one for the domain you care about. Plenty of confusing &#8220;the certificate is for the wrong domain&#8221; reports come from testing without SNI.</p>



<p class="wp-block-paragraph">Three parts of the output are worth reading carefully:</p>



<ul class="wp-block-list">
<li><strong>The &#8220;Certificate chain&#8221; block</strong> near the top, listing each certificate the server sent with its subject and issuer.</li>

<li><strong>The &#8220;Verify return code&#8221;</strong> at the bottom. <code>0 (ok)</code> is what you want. <code>20</code> and <code>21</code> both point at chain problems.</li>

<li><strong>The subject line of the first certificate</strong>, which tells you whether you even got the certificate you expected.</li>
</ul>



<p class="wp-block-paragraph">If you want the certificate details without the chain noise, pipe it into <code>x509</code>:</p>



<pre class="wp-block-code"><code>echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null 
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName</code></pre>



<p class="wp-block-paragraph">That single line answers &#8220;who issued it&#8221;, &#8220;when does it expire&#8221;, and &#8220;what hostnames does it cover&#8221; in one shot. It is the first thing I run.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family one: certificate chain issues</h2>



<p class="wp-block-paragraph">A publicly trusted certificate is almost never signed directly by a root. There is at least one intermediate CA in between. The client already trusts the root — it ships in the operating system or browser trust store — but it has no idea about the intermediate unless you hand it over. Your server&#8217;s job is to send the leaf certificate plus every intermediate needed to walk up to a trusted root.</p>



<h3 class="wp-block-heading">The classic mistake: serving the leaf without the intermediates</h3>



<p class="wp-block-paragraph">If you use Let&#8217;s Encrypt, certbot writes several files into <code>/etc/letsencrypt/live/example.com/</code>. Two of them look interchangeable and are not:</p>



<ul class="wp-block-list">
<li><code>cert.pem</code> — the leaf certificate on its own.</li>

<li><code>fullchain.pem</code> — the leaf plus the intermediates.</li>

<li><code>chain.pem</code> — the intermediates on their own.</li>

<li><code>privkey.pem</code> — the private key.</li>
</ul>



<p class="wp-block-paragraph">Nginx wants <code>fullchain.pem</code>. Point it at <code>cert.pem</code> and browsers will still work, so the mistake survives testing and surfaces weeks later when an API client fails.</p>



<pre class="wp-block-code"><code>server {
    listen 443 ssl;
    server_name example.com www.example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}</code></pre>



<p class="wp-block-paragraph">On Apache 2.4.8 and later, <code>SSLCertificateFile</code> can hold the leaf and the intermediates concatenated together, and <code>SSLCertificateChainFile</code> is deprecated. If you inherited a config that still uses the old directive, it will work, but it is worth cleaning up:</p>



<pre class="wp-block-code"><code>SSLCertificateFile    /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem</code></pre>



<h3 class="wp-block-heading">Wrong order, or a root that should not be there</h3>



<p class="wp-block-paragraph">Order is part of the specification, not a suggestion. The leaf comes first, then each issuer in sequence walking upward. Some clients tolerate a shuffled bundle; several do not, and the ones that do not tend to be the ones you find out about from an angry customer.</p>



<p class="wp-block-paragraph">Including the root certificate at the end is a different case: it is harmless from a trust perspective, since the client ignores it and uses its own copy, but it adds bytes to every single handshake for no benefit. Leave it out.</p>



<h3 class="wp-block-heading">Verifying a chain offline</h3>



<p class="wp-block-paragraph">You can check a bundle before deploying it, which is far better than discovering the problem in production:</p>



<pre class="wp-block-code"><code># Does the leaf verify against the intermediates you plan to serve?
openssl verify -untrusted chain.pem cert.pem

# Print the subject and issuer of every certificate in a bundle, in order
openssl crl2pkcs7 -nocrl -certfile fullchain.pem 
  | openssl pkcs7 -print_certs -noout</code></pre>



<p class="wp-block-paragraph">That second command is the one I reach for most. Read the output top to bottom: the issuer of each certificate should be the subject of the next one down. The moment that pattern breaks, you have found your gap.</p>



<h3 class="wp-block-heading">The key and the certificate do not match</h3>



<p class="wp-block-paragraph">Less common, but it produces a spectacularly unhelpful error. After a renewal where files were copied around by hand, confirm the pair still belongs together — the two hashes must be identical:</p>



<pre class="wp-block-code"><code>openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa  -noout -modulus -in privkey.pem | openssl md5</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family two: SAN mismatches</h2>



<p class="wp-block-paragraph">The Common Name field is dead. It has been deprecated for hostname matching for years, and Chrome stopped honouring it entirely back in 2017. What matters now is the Subject Alternative Name extension. If the hostname a client requested is not in the SAN list, the connection fails, regardless of what the CN says.</p>



<p class="wp-block-paragraph">Check what a certificate actually covers:</p>



<pre class="wp-block-code"><code># From a file
openssl x509 -noout -ext subjectAltName -in cert.pem

# From a live server
echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null 
  | openssl x509 -noout -ext subjectAltName</code></pre>



<h3 class="wp-block-heading">The four SAN mistakes that keep recurring</h3>



<ul class="wp-block-list">
<li><strong>Apex without www, or www without apex.</strong> These are two distinct names. A certificate for <code>example.com</code> does not cover <code>www.example.com</code>. Both need to be in the SAN list, and both need to be passed to certbot with separate <code>-d</code> flags.</li>

<li><strong>Assuming a wildcard covers everything.</strong> It does not. <code>*.example.com</code> matches exactly one label. It covers <code>api.example.com</code>. It does not cover <code>example.com</code> itself, and it does not cover <code>staging.api.example.com</code>. Multi-level subdomains need their own wildcard or their own entry.</li>

<li><strong>A new subdomain that never made it into the renewal.</strong> You add <code>metrics.example.com</code> to nginx, point DNS at it, and forget that the certificate was issued for a fixed list of names. Adding a name means reissuing, not just reloading.</li>

<li><strong>Connecting by IP address.</strong> Almost no public certificate includes an IP SAN, so hitting the server by IP will always fail hostname verification. That is expected behaviour, not a broken certificate.</li>
</ul>



<p class="wp-block-paragraph">One more that catches people out: if you use Cloudflare in proxied mode, the certificate a visitor sees is Cloudflare&#8217;s edge certificate, not yours. Debugging the origin means bypassing the proxy and testing the origin IP directly with the <code>Host</code> header set correctly, or temporarily turning the orange cloud grey.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family three: expiry, and why automation is no longer optional</h2>



<p class="wp-block-paragraph">Expiry used to be a calendar problem. You bought a certificate for a year, set a reminder, and dealt with it once. That era is closing.</p>



<p class="wp-block-paragraph">In April 2025 the CA/Browser Forum passed ballot SC-081v3, which phases the maximum validity period for publicly trusted TLS certificates down from 398 days to 47 days. The schedule runs in three steps: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029. The first phase is already in force. Domain validation reuse windows shrink alongside it.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The 47-day target was chosen deliberately: short enough that manual renewal stops being viable at any meaningful scale. Automation is the point of the change, not a side effect of it.</p>
</blockquote>



<p class="wp-block-paragraph">If you run a handful of domains, this is a nuisance. If you run fifty, manual renewal is already finished as a strategy.</p>



<h3 class="wp-block-heading">Getting ACME renewal right</h3>



<p class="wp-block-paragraph">Certbot installs a systemd timer or cron entry that attempts renewal twice a day and does nothing unless the certificate is close to expiry. Confirm it is actually there:</p>



<pre class="wp-block-code"><code>systemctl list-timers | grep certbot
certbot renew --dry-run</code></pre>



<p class="wp-block-paragraph">The dry run exercises the whole path against the staging environment without touching rate limits. Run it after any change to your web server config, not just after installing certbot. A renewal that worked last quarter can break because someone added a redirect that intercepts the ACME challenge path.</p>



<h3 class="wp-block-heading">The trap: renewal succeeds, the site still serves the old certificate</h3>



<p class="wp-block-paragraph">This is the failure I see most often, and it is genuinely maddening because every log says success. Certbot writes the new certificate to disk. Nginx has the old one loaded in memory and keeps serving it until it is told otherwise. The certificate on disk is valid, the certificate on the wire is expired, and everything looks fine from the server&#8217;s point of view.</p>



<p class="wp-block-paragraph">The fix is a deploy hook, which runs only when a certificate was actually renewed:</p>



<pre class="wp-block-code"><code>certbot renew --deploy-hook "systemctl reload nginx"</code></pre>



<p class="wp-block-paragraph">Better still, make it permanent by adding it to the renewal configuration at <code>/etc/letsencrypt/renewal/example.com.conf</code>, under the <code>[renewalparams]</code> section, so it survives however the renewal happens to be triggered:</p>



<pre class="wp-block-code"><code>renew_hook = systemctl reload nginx</code></pre>



<p class="wp-block-paragraph">Use <code>reload</code> rather than <code>restart</code>. A reload picks up the new certificate without dropping connections.</p>



<p class="wp-block-paragraph">Remember that anything else terminating TLS needs the same treatment. A mail server, a database proxy, a Docker container with the certificate baked into its image — each one needs its own reload step, and containers in particular will happily run for months with an expired certificate mounted from the host.</p>



<h3 class="wp-block-heading">Wildcards need DNS-01</h3>



<p class="wp-block-paragraph">Wildcard certificates cannot be issued through HTTP-01 validation. They require DNS-01, which means certbot needs to create a TXT record on your domain, which means API credentials for your DNS provider. Use a provider plugin rather than the manual mode — manual DNS-01 cannot be automated, and an unautomated wildcard on a 47-day cycle is an outage with a scheduled start time.</p>



<p class="wp-block-paragraph">If issuance fails outright with a policy error, check your CAA records before assuming the ACME client is broken. A CAA record that names a different CA will block issuance entirely:</p>



<pre class="wp-block-code"><code>dig CAA example.com +short</code></pre>



<h3 class="wp-block-heading">Monitor from outside, not from the server</h3>



<p class="wp-block-paragraph">Checking the certificate file on disk tells you what should be served. Only an external check tells you what is being served. Given the reload trap above, that difference is the entire point.</p>



<p class="wp-block-paragraph">A minimal check that works anywhere:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# Usage: ./cert-days.sh example.com
# Exits non-zero if fewer than 21 days remain.
set -euo pipefail

DOMAIN="$1"
THRESHOLD=21

END=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "${DOMAIN}" 2&gt;/dev/null 
      | openssl x509 -noout -enddate | cut -d= -f2)

DAYS=$(( ( $(date -d "${END}" +%s) - $(date +%s) ) / 86400 ))
echo "${DOMAIN}: ${DAYS} days remaining (expires ${END})"

[ "${DAYS}" -lt "${THRESHOLD}" ] &amp;&amp; exit 1
exit 0</code></pre>



<p class="wp-block-paragraph">If you already run Prometheus, the blackbox exporter gives you this for free through the <code>probe_ssl_earliest_cert_expiry</code> metric. An alert rule looks like this:</p>



<pre class="wp-block-code"><code>groups:
  - name: tls
    rules:
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 &lt; 21
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "TLS certificate expiring in under 21 days"</code></pre>



<p class="wp-block-paragraph">Set the threshold well above your renewal window. With 200-day certificates renewing at 30 days remaining, a 21-day alert gives you nine days of warning that something in the pipeline broke. Once 47-day certificates arrive, those numbers compress hard, and an alert that fires too late is the same as no alert.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A workflow for diagnosing TLS certificate errors</h2>



<p class="wp-block-paragraph">When something breaks, work through it in this order. It takes about a minute and rules out whole families of causes as you go.</p>



<ol class="wp-block-list">
<li><strong>Reproduce outside the browser.</strong> Use <code>curl -v</code> or <code>openssl s_client</code>. If the browser works and curl does not, you are almost certainly looking at a chain problem.</li>

<li><strong>Read the verify return code.</strong> Code 20 or 21 means a chain issue. A hostname mismatch message means a SAN issue. A date error means expiry — or a wrong system clock on the client.</li>

<li><strong>Print the dates.</strong> Rules expiry in or out in one command, and tells you whether the certificate on the wire matches the one on disk.</li>

<li><strong>Print the SANs.</strong> Confirm the exact hostname being requested appears in the list. Watch for apex versus www.</li>

<li><strong>Walk the chain.</strong> Each certificate&#8217;s issuer should be the next one&#8217;s subject. Find where the sequence breaks.</li>

<li><strong>Compare disk against wire.</strong> If the file is newer than what is being served, you have a reload problem, not a certificate problem.</li>

<li><strong>Check the client&#8217;s trust store.</strong> If everything above is clean and one specific client still fails, the problem has moved to their end — an old Android device, a Java service with its own <code>cacerts</code>, or a container image built on a base with a stale CA bundle.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes worth avoiding</h2>



<ul class="wp-block-list">
<li>Testing only in a browser, and mistaking its silent chain repair for a working configuration.</li>

<li>Pointing the web server at <code>cert.pem</code> instead of <code>fullchain.pem</code>.</li>

<li>Renewing without reloading the service that holds the certificate in memory.</li>

<li>Running <code>certbot renew</code> on a schedule but never running <code>--dry-run</code> after config changes.</li>

<li>Copying key and certificate files between servers by hand and losing track of which pair belongs together.</li>

<li>Monitoring the file on disk rather than the certificate actually being served.</li>

<li>Forgetting that mail servers, load balancers, and containers terminate TLS too, and need their own renewal path.</li>

<li>Adding a subdomain to the web server config and assuming the existing certificate covers it.</li>
</ul>



<h2 class="wp-block-heading">Best practices that hold up over time</h2>



<ul class="wp-block-list">
<li>Automate issuance and renewal from day one, even for a single domain. The habit matters more than the scale.</li>

<li>Always use a deploy hook, and always reload rather than restart.</li>

<li>Alert on days remaining, measured externally, with a threshold at least twice your renewal window.</li>

<li>Keep an inventory of every hostname you hold a certificate for. You cannot monitor what you have forgotten about.</li>

<li>Prefer DNS-01 with a provider plugin if you need wildcards, and never rely on manual DNS validation.</li>

<li>Set CAA records deliberately so you know which CAs are permitted to issue for your domain.</li>

<li>Test with a non-browser client before calling a certificate deployment finished.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why does my site work in Chrome but fail in curl?</h3>



<p class="wp-block-paragraph">Almost always an incomplete chain. Browsers fetch missing intermediate certificates automatically using the Authority Information Access extension; curl does not. Point your web server at the full chain file rather than the leaf certificate and both will work.</p>



<h3 class="wp-block-heading">What does &#8220;unable to get local issuer certificate&#8221; actually mean?</h3>



<p class="wp-block-paragraph">The client could not build a path from your certificate to a root it trusts. Nine times out of ten the server is not sending its intermediates. Occasionally the client&#8217;s own CA bundle is out of date, which is common in older container base images.</p>



<h3 class="wp-block-heading">Does a wildcard certificate cover the root domain?</h3>



<p class="wp-block-paragraph">No. <code>*.example.com</code> matches a single label, so it covers <code>www.example.com</code> but not <code>example.com</code> and not <code>a.b.example.com</code>. Include the apex explicitly as an additional SAN entry when you request the certificate.</p>



<h3 class="wp-block-heading">My certificate renewed but the site still shows it as expired. Why?</h3>



<p class="wp-block-paragraph">The service holding the certificate has not reloaded. The new file is on disk; the old one is still in memory. Add a deploy hook that reloads the service on renewal, then verify from outside with <code>openssl s_client</code> rather than trusting the file timestamp.</p>



<h3 class="wp-block-heading">How short will certificate lifetimes actually get?</h3>



<p class="wp-block-paragraph">Under the CA/Browser Forum schedule, 200 days applies from March 2026, 100 days from March 2027, and 47 days from March 2029. Private and internal CAs are not bound by these rules, but public certificates are.</p>



<h3 class="wp-block-heading">Is the Common Name field still used at all?</h3>



<p class="wp-block-paragraph">Not for hostname validation. Modern clients read only the Subject Alternative Name extension. A certificate whose CN matches but whose SAN list does not include the hostname will be rejected.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">TLS certificate errors feel opaque mostly because the error text points at the wrong layer. Once you sort them into chain, SAN, or expiry, each one has a short diagnostic path and an obvious fix. Get comfortable with <code>openssl s_client</code> and <code>openssl x509</code> and you will resolve most incidents faster than it takes to find the right dashboard.</p>



<p class="wp-block-paragraph">The expiry category is the one worth engineering away permanently. With maximum lifetimes dropping toward 47 days, any process that depends on a person remembering something will fail eventually. Automate the renewal, add the deploy hook, and alert on what is actually being served rather than what is sitting on disk. Do that once and TLS certificate errors go back to being a rare curiosity instead of a recurring outage.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with TLS, certificates, or infrastructure automation?</h2>



<p class="wp-block-paragraph">I work with teams on exactly this kind of problem — broken certificate chains, ACME automation across multiple servers, expiry monitoring wired into Prometheus and Grafana, and the wider DevOps plumbing that keeps it all running quietly in the background.</p>



<p class="wp-block-paragraph">If you would rather hand this off than debug it at two in the morning, you can hire me on Upwork.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/tls-certificate-errors/">TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
