<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>OpenSSL | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/openssl/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/openssl/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 19 Sep 2026 14:03:05 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>OpenSSL | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/openssl/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</title>
		<link>https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:23 +0000</pubDate>
				<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[ACME]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Certificate Chain]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[DNS]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OpenSSL]]></category>
		<category><![CDATA[SSL Certificate]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=64</guid>

					<description><![CDATA[<p>Your browser shows a padlock and your partner's API client throws "unable to get local issuer certificate". Both are correct. A practical guide to the four things a TLS client actually checks, and to renewal automation that reaches the running server instead of just the disk.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/">It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A partner emails to say your API has been rejecting their integration for two days. Their client throws <code>unable to get local issuer certificate</code>. You open the URL in Chrome, see a padlock, and reply that it works fine on your end. Then someone runs <code>curl</code> from a clean container and it fails there too.</p>



<p class="wp-block-paragraph">Your server is only sending the leaf certificate. The intermediate that links it to a trusted root is missing. Your browser papers over that: it either has the intermediate cached from another site, or it quietly fetches it using the AIA extension in the certificate. <code>curl</code>, Java, Go, Python and every other non-browser client do no such thing. They see a certificate signed by something they do not trust and stop.</p>



<p class="wp-block-paragraph">This is the most common of all <strong>TLS certificate errors</strong> and it is close to invisible from a desktop. The padlock in your browser is not a test. It is a client with a large safety net doing you a favour.</p>



<p class="wp-block-paragraph">This post covers the four things a client actually checks and how each one fails: the chain, the names, the clock, and the trust store. Then the automation half, which has its own quiet failure: a renewal that succeeds and never reaches the running server. Plus the two things about TLS that have genuinely changed recently and will break config you copied from an older guide.</p>



<h2 class="wp-block-heading">Look at what the server is actually sending</h2>



<p class="wp-block-paragraph">Before theorising, get the facts. One command tells you almost everything.</p>



<pre class="wp-block-code"><code># -servername sends SNI. Without it you get whatever the default
# vhost serves, which is a different certificate on most servers
# and the source of a lot of confusing results.
# -showcerts prints every certificate the server sent, in order.
openssl s_client -connect example.com:443 -servername example.com -showcerts &lt;/dev/null</code></pre>



<p class="wp-block-paragraph">Read the top of that output. There is a <code>Certificate chain</code> block listing what arrived, numbered from zero, with <code>s:</code> for subject and <code>i:</code> for issuer. Entry 0 is your certificate. Entry 1 should be the intermediate that issued it, and its subject should exactly match entry 0&#8217;s issuer.</p>



<p class="wp-block-paragraph">If the chain block has only entry 0, that is your bug. Right at the bottom, <code>Verify return code</code> gives the verdict: <code>0 (ok)</code> is what you want, <code>21 (unable to verify the first certificate)</code> means the chain is incomplete, and <code>20 (unable to get local issuer certificate)</code> means the issuer is not in the local trust store.</p>



<p class="wp-block-paragraph">Run it from a machine that is not yours. A clean container is ideal, because your laptop has accumulated intermediates and corporate roots that a customer&#8217;s server has not.</p>



<h2 class="wp-block-heading">Chain problems</h2>



<p class="wp-block-paragraph">Almost all of these come down to one file choice. ACME clients write out several files and two of them look interchangeable:</p>



<ul class="wp-block-list">
<li><strong><code>cert.pem</code></strong> is the leaf alone. Using this is the mistake.</li>
<li><strong><code>chain.pem</code></strong> is the intermediates alone.</li>
<li><strong><code>fullchain.pem</code></strong> is leaf plus intermediates, in the right order. This is the one your server should point at.</li>
</ul>



<pre class="wp-block-code"><code># nginx
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;</code></pre>



<pre class="wp-block-code"><code># Apache. On current versions the chain goes in SSLCertificateFile.
# SSLCertificateChainFile is deprecated; guides that still use it
# are old enough that other advice in them is probably stale too.
SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem</code></pre>



<p class="wp-block-paragraph">Order matters as well as contents. The leaf must come first, then each intermediate that signs the one above it. A chain assembled in the wrong order will be rejected by strict clients even though every certificate in it is correct. You can check a chain offline before deploying it:</p>



<pre class="wp-block-code"><code># Does this leaf chain to a trusted root using these intermediates?
# Exits non-zero and explains itself if not.
openssl verify -untrusted chain.pem cert.pem</code></pre>



<p class="wp-block-paragraph">Do not include the root certificate in the file you serve. The client either has it or does not, and sending it wastes bytes on every handshake without changing any outcome.</p>



<h2 class="wp-block-heading">Name problems</h2>



<p class="wp-block-paragraph">The second family. A certificate can be perfectly valid and still be the wrong certificate for the name being requested.</p>



<pre class="wp-block-code"><code># The hostname the client asked for must appear in here.
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"</code></pre>



<p class="wp-block-paragraph">Three things people get wrong here:</p>



<ul class="wp-block-list">
<li><strong>The Common Name does not count.</strong> Modern clients ignore the CN field entirely for hostname matching. If the name is only in the CN and not in the SAN list, it will fail, and the error message will not explain that clearly.</li>
<li><strong>Wildcards are exactly one label deep.</strong> <code>*.example.com</code> covers <code>www.example.com</code> and <code>api.example.com</code>. It does not cover <code>example.com</code> itself, and it does not cover <code>a.b.example.com</code>. The bare domain has to be listed separately, and it usually is not.</li>
<li><strong>SNI decides which certificate you get.</strong> One IP serving many sites picks the certificate based on the name the client sends during the handshake. A client that does not send SNI, or an <code>openssl</code> command without <code>-servername</code>, gets the default vhost&#8217;s certificate and produces a name mismatch that has nothing to do with your actual configuration.</li>
</ul>



<h2 class="wp-block-heading">Time and trust problems</h2>



<pre class="wp-block-code"><code># Local file
openssl x509 -in cert.pem -noout -dates

# Live endpoint, subject and dates in one go
openssl s_client -connect example.com:443 -servername example.com &lt;/dev/null 2&gt;/dev/null 
  | openssl x509 -noout -dates -subject</code></pre>



<p class="wp-block-paragraph">Expired is obvious. <code>notBefore</code> in the future is the interesting one, because it almost never means the certificate is wrong. It means the client&#8217;s clock is wrong. Containers with no NTP, a VM restored from a snapshot, an embedded device with a dead battery: all of them produce &#8220;certificate not yet valid&#8221; against a perfectly good certificate. Check the clock on the failing machine before you touch anything on the server.</p>



<p class="wp-block-paragraph">Trust store problems look similar but come from a different place. The root that signed your chain has to be in the trust store of the machine making the request, and old systems have old trust stores. A device that has not received an OS update in years may simply not know about a certificate authority that appeared since. There is no server-side fix for that beyond choosing a CA with broader legacy coverage, which is one of the few genuine arguments for a paid certificate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The renewal that succeeded and changed nothing</h2>



<p class="wp-block-paragraph">Here is the second invisible failure, and it catches experienced people. Your renewal timer runs, the ACME client obtains a new certificate, writes it to disk, logs success. The site goes down anyway.</p>



<p class="wp-block-paragraph">Nginx read the certificate into memory when it started. It does not watch the file. Neither does Apache, HAProxy, Postfix, or almost anything else. Until the service is reloaded it keeps serving the certificate it loaded at boot, which is the one that just expired. Everything in your logs says the renewal worked, because it did. The delivery step was never wired up.</p>



<p class="wp-block-paragraph">Certbot handles this with deploy hooks, which run only when a certificate was actually renewed. Put a script in the hooks directory and it applies to every certificate on the box:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-services.sh
# Remember: chmod +x, or it silently never runs.
set -euo pipefail

# Validate the config BEFORE reloading. A reload with a broken
# config can take the site down at the exact moment you were
# trying to keep it up.
nginx -t
systemctl reload nginx

# Anything else that holds a copy of the certificate.
# systemctl reload postfix</code></pre>



<p class="wp-block-paragraph">Then prove the whole path works without waiting sixty days to find out:</p>



<pre class="wp-block-code"><code># Exercises the full renewal against the staging environment.
# Installs nothing, counts against no rate limits, and will
# fail loudly if your challenge setup is broken.
certbot renew --dry-run</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>--dry-run</code> does not run deploy hooks by default, precisely because nothing was deployed. So it validates the issuance path but not the reload. Test that separately by touching the service and confirming the served certificate changes.</p>



<h2 class="wp-block-heading">Expiry automation, and why it is no longer optional</h2>



<p class="wp-block-paragraph">This part has a date attached, so it is worth stating plainly rather than dancing around it.</p>



<p class="wp-block-paragraph">The CA/Browser Forum approved a phased reduction of maximum certificate lifetimes. The cap dropped from 398 days to 200 days on 15 March 2026, falls to 100 days on 15 March 2027, and reaches 47 days on 15 March 2029. Domain validation reuse periods shrink alongside it.</p>



<p class="wp-block-paragraph">The practical consequence: any process that involves a person remembering to renew something once a year is already broken, and by the third phase you are renewing every six weeks. Whatever you were getting away with manually, stop. ACME with a timer is the baseline. Certbot is the default choice, acme.sh is lighter and easier to script around, and Caddy obtains and renews certificates with no configuration at all, which is a genuinely good reason to consider it for a simple reverse proxy.</p>



<h3 class="wp-block-heading">Choosing a challenge type</h3>



<ul class="wp-block-list">
<li><strong>HTTP-01</strong> proves control by serving a file over port 80. Simple, and it breaks whenever port 80 is closed, redirected wholesale to HTTPS in a way that mangles the challenge path, or sitting behind a proxy that does not forward it.</li>
<li><strong>DNS-01</strong> proves control by writing a TXT record. More setup, needs an API token for your DNS provider, and it is the only option for wildcard certificates. It also works for hosts with no inbound internet access at all, which is why I reach for it on anything internal.</li>
</ul>



<p class="wp-block-paragraph">The trade-off with DNS-01 is a credential: your renewal process now holds a token that can edit your DNS zone. Scope it to the narrowest permission your provider offers, and if they support delegating just the <code>_acme-challenge</code> records to a separate zone, do that instead.</p>



<h3 class="wp-block-heading">Monitor expiry externally</h3>



<p class="wp-block-paragraph">Do not rely on the renewal system reporting its own health. If it is broken, it is also probably not telling you. Let&#8217;s Encrypt has discontinued its expiration reminder emails, so if that was quietly acting as your safety net, it is not one any more.</p>



<p class="wp-block-paragraph">Check from outside, against the live endpoint, the way a user would:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

HOST="example.com"
DAYS=21

# -checkend takes a number of seconds and exits non-zero if the
# certificate expires within that window. No parsing dates by hand.
if ! openssl s_client -connect "$HOST:443" -servername "$HOST" &lt;/dev/null 2&gt;/dev/null 
     | openssl x509 -noout -checkend $(( DAYS * 86400 )); then
  echo "TLS certificate for $HOST expires within $DAYS days" &gt;&amp;2
  exit 1
fi</code></pre>



<p class="wp-block-paragraph">If you already run Prometheus, the blackbox exporter gives you <code>probe_ssl_earliest_cert_expiry</code> for free, which is a timestamp you can alert on and graph in Grafana across every endpoint you own. That is the better answer at more than a handful of hosts, because it also catches the certificates nobody remembered existed.</p>



<h2 class="wp-block-heading">Two pieces of config that are now wrong</h2>



<p class="wp-block-paragraph"><strong>OCSP stapling with Let&#8217;s Encrypt.</strong> Let&#8217;s Encrypt removed the OCSP responder URL from newly issued certificates in May 2025 and shut down its responders in August 2025. If your config still has this, it does nothing except print a warning on every reload:</p>



<pre class="wp-block-code"><code># nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate
# Comment these out if your certificates come from Let's Encrypt.
# ssl_stapling on;
# ssl_stapling_verify on;</code></pre>



<p class="wp-block-paragraph">Nothing breaks by leaving them, but a warning that appears on every reload is a warning people learn to ignore, and then they ignore the next one too.</p>



<p class="wp-block-paragraph"><strong>Hand-tuned cipher suite lists.</strong> If you pasted a <code>ssl_ciphers</code> line from a guide more than a couple of years old, re-check it against the current Mozilla SSL Configuration Generator. Lists that were considered hardened at the time now sometimes exclude things you want or include things you do not, and the defaults have improved considerably.</p>



<h2 class="wp-block-heading">When there is a CDN in front</h2>



<p class="wp-block-paragraph">Behind Cloudflare or a similar edge, there are two separate TLS connections: browser to edge, and edge to your origin. They use different certificates and fail independently.</p>



<p class="wp-block-paragraph">This creates a specific blind spot. Your origin certificate can be expired, self-signed or serving the wrong name, and visitors will see nothing wrong, because the edge presents its own valid certificate and, in the more permissive encryption modes, does not verify the origin. It works until you switch to strict origin validation, or until the edge is bypassed, and then everything fails at once for reasons that look unrelated to anything you changed.</p>



<p class="wp-block-paragraph">Test the origin directly by name, bypassing DNS. If your origin is a VPS with a provider like InterServer and the public DNS points at the CDN, this is the only way to see what the origin is really serving:</p>



<pre class="wp-block-code"><code># --resolve forces the hostname to a specific IP without touching
# /etc/hosts, so SNI and Host header stay correct.
curl -v --resolve example.com:443:203.0.113.10 https://example.com/</code></pre>



<h2 class="wp-block-heading">Troubleshooting by error message</h2>



<h3 class="wp-block-heading">unable to get local issuer certificate</h3>



<p class="wp-block-paragraph">Missing intermediate on the server, or a trust store on the client that lacks the root. Check the server first with <code>-showcerts</code>; if the chain is complete, the problem is on the client side.</p>



<h3 class="wp-block-heading">certificate has expired, but I just renewed it</h3>



<p class="wp-block-paragraph">The service was never reloaded. Compare the dates on disk with the dates being served. If they differ, that is your answer, and the fix is a deploy hook rather than another manual renewal.</p>



<h3 class="wp-block-heading">Hostname mismatch on a name that is definitely in the certificate</h3>



<p class="wp-block-paragraph">Either SNI is not reaching the right vhost, or the name is in the Common Name rather than the SAN list, or you are hitting a wildcard at the wrong depth. Check the SAN list and re-run <code>s_client</code> with an explicit <code>-servername</code>.</p>



<h3 class="wp-block-heading">Works over IPv4, fails over IPv6 (or the reverse)</h3>



<p class="wp-block-paragraph">Two different machines are answering, and only one of them was updated. Force the family with <code>curl -4</code> and <code>curl -6</code> and compare. Load balancers with one member out of sync produce the same intermittent pattern.</p>



<h3 class="wp-block-heading">Renewal fails with a challenge error</h3>



<p class="wp-block-paragraph">For HTTP-01, confirm that <code>http://example.com/.well-known/acme-challenge/test</code> is reachable from outside without being redirected or rewritten. A blanket redirect to HTTPS is the usual culprit. For DNS-01, check that the TXT record actually appears with <code>dig</code>, and give slow-propagating providers more time.</p>



<h3 class="wp-block-heading">Everything looks right and it still fails</h3>



<p class="wp-block-paragraph">Run an external scan. SSL Labs or a local run of <code>testssl.sh</code> will check things you did not think to, including protocol support and chain issues from the perspective of clients you do not have installed.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing the server at <code>cert.pem</code> instead of <code>fullchain.pem</code>.</li>
<li>Treating a padlock in your own browser as proof the certificate is correctly deployed.</li>
<li>Testing with <code>openssl s_client</code> without <code>-servername</code> and debugging the wrong certificate.</li>
<li>Assuming <code>*.example.com</code> covers the bare domain.</li>
<li>Putting the hostname only in the Common Name.</li>
<li>Renewing without reloading the service.</li>
<li>Forgetting <code>chmod +x</code> on a deploy hook, so it silently never runs.</li>
<li>Relying on the renewal system to tell you when it is broken.</li>
<li>Including the root certificate in the served chain.</li>
<li>Leaving OCSP stapling enabled with certificates that no longer support it.</li>
<li>Ignoring the origin certificate because the CDN hides its state.</li>
<li>Debugging a &#8220;not yet valid&#8221; certificate on the server when the client&#8217;s clock is wrong.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Serve the full chain, leaf first, without the root.</li>
<li>Verify from a clean machine or container, never from the laptop you browse on.</li>
<li>Put every name you serve in the SAN list, including the bare domain alongside any wildcard.</li>
<li>Automate renewal with ACME and a timer, and run <code>certbot renew --dry-run</code> after any change to the setup.</li>
<li>Attach a deploy hook that validates the config before reloading, and make it executable.</li>
<li>Monitor expiry externally against the live endpoint, and alert with weeks of headroom.</li>
<li>Use DNS-01 for wildcards and internal hosts, with a narrowly scoped API token.</li>
<li>Keep NTP running everywhere, including inside containers.</li>
<li>Test the origin directly when there is a CDN in front, not just the public hostname.</li>
<li>Inventory your certificates, because the one that takes you down is the one nobody remembered.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Why does my certificate work in Chrome but fail in curl?</h3>



<p class="wp-block-paragraph">Your server is not sending the intermediate certificate. Browsers work around this by caching intermediates from other sites and by fetching them from the URL in the certificate&#8217;s AIA extension. Command-line tools and application HTTP libraries do neither. Switch the server to <code>fullchain.pem</code> and reload.</p>



<h3 class="wp-block-heading">What is the difference between cert.pem, chain.pem and fullchain.pem?</h3>



<p class="wp-block-paragraph">Leaf only, intermediates only, and both concatenated in the correct order. Web servers want <code>fullchain.pem</code>. The private key stays in <code>privkey.pem</code> and never goes in any of them.</p>



<h3 class="wp-block-heading">Does a wildcard certificate cover the root domain?</h3>



<p class="wp-block-paragraph">No. <code>*.example.com</code> matches exactly one label, so it covers <code>www.example.com</code> but not <code>example.com</code> and not <code>a.b.example.com</code>. Request the bare domain as an additional SAN, which most ACME clients will do if you pass both names.</p>



<h3 class="wp-block-heading">How often should certificates renew now?</h3>



<p class="wp-block-paragraph">Whatever your CA issues, renew at roughly two-thirds of the lifetime so a failed attempt has room to retry before anything expires. Since maximum lifetimes are on a published downward schedule, build for automated renewal at any interval rather than tuning a number you will have to change again.</p>



<h3 class="wp-block-heading">Should I still enable OCSP stapling?</h3>



<p class="wp-block-paragraph">Not with Let&#8217;s Encrypt certificates, which no longer carry a responder URL. If your CA still supports it, stapling remains worthwhile. Check whether your certificate actually contains an OCSP URL before enabling the directives, rather than assuming.</p>



<h3 class="wp-block-heading">Is a paid certificate more secure than a free one?</h3>



<p class="wp-block-paragraph">Cryptographically, no. The encryption is identical. What money buys is broader compatibility with old trust stores, longer validation reuse, warranties, and human support when something goes wrong. For a normal public website, a free automated certificate is the right default.</p>



<h3 class="wp-block-heading">How do I get a certificate for something with no public DNS or open ports?</h3>



<p class="wp-block-paragraph">DNS-01, on a public domain you control, pointed at whatever internal address you like. The CA only needs to see the TXT record; it never needs to reach the host. This is how internal services get publicly trusted certificates without being exposed.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Most TLS certificate errors are not about cryptography. They are about delivery: whether the server sends everything the client needs, whether the name it sends matches the name that was asked for, and whether the certificate sitting in memory is the same one sitting on disk.</p>



<p class="wp-block-paragraph">So stop checking with your browser. Check with <code>openssl s_client</code> from a machine that owes you no favours, and make sure something outside your server is watching the expiry date. As lifetimes keep shrinking, the gap between a renewal process that works and one that merely appears to work gets narrower every year.</p>



<h2 class="wp-block-heading">Certificates causing you trouble?</h2>



<p class="wp-block-paragraph">Certificate problems tend to arrive at bad moments and turn out to be one wrong file path. Things I help with:</p>



<ul class="wp-block-list">
<li>Diagnosing a certificate that works for some clients and not others, and telling you exactly which link in the chain is missing.</li>
<li>Setting up ACME renewal with deploy hooks across nginx, Apache, HAProxy, mail servers and anything else holding a copy.</li>
<li>Migrating to DNS-01 for wildcards and internal hosts, with properly scoped API credentials.</li>
<li>Certificate inventory and expiry monitoring in Prometheus and Grafana, so nothing expires without warning.</li>
<li>Preparing a certificate estate for shorter lifetimes, replacing anything that still depends on a person remembering.</li>
<li>Origin and edge TLS configuration behind a CDN, including strict origin validation.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>openssl s_client -connect yourhost:443 -servername yourhost -showcerts</code> and the error your failing client reports, and I will tell you what is wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/debugging-tls-certificate-errors/">It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</title>
		<link>https://john-nessime.com/blog/system-administration/tls-certificate-errors/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 30 Jul 2026 15:02:33 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OpenSSL]]></category>
		<category><![CDATA[SSL Certificate]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=31</guid>

					<description><![CDATA[<p>Almost every TLS certificate error traces back to one of three causes: a broken chain, a hostname the certificate does not cover, or an expiry nobody was watching. Here is how to tell them apart in under a minute, and how to stop the third one from ever happening again.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/tls-certificate-errors/">TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The site loads fine in your browser. Then a mobile app starts throwing handshake failures, a webhook from a payment provider stops arriving, and someone on the team runs <code>curl</code> and gets &#8220;SSL certificate problem: unable to get local issuer certificate&#8221;. Nothing changed on the server. Except something did, and your browser has been quietly covering for it.</p>



<p class="wp-block-paragraph">Nearly every TLS certificate error I have had to chase down comes back to one of three things: the chain your server sends is incomplete, the certificate does not actually cover the hostname being requested, or it expired and no alert fired. This article walks through all three — how to identify which one you are looking at, the exact commands to prove it, and how to set up renewal automation so the third category stops being a category at all.</p>



<h2 class="wp-block-heading">Why TLS certificate errors are so confusing to debug</h2>



<p class="wp-block-paragraph">Two things conspire to make this harder than it should be.</p>



<p class="wp-block-paragraph">The first is that browsers are forgiving in ways other clients are not. When a server sends an incomplete certificate chain, most modern browsers will quietly fetch the missing intermediate using the Authority Information Access extension embedded in the certificate. They fix your misconfiguration on the fly and show you a padlock. Meanwhile <code>curl</code>, Python&#8217;s <code>requests</code>, Java clients, Go services, and older Android devices do no such thing. They fail. This is why &#8220;it works in Chrome&#8221; is worth almost nothing as a diagnostic signal.</p>



<p class="wp-block-paragraph">The second is that error messages describe the symptom rather than the cause. &#8220;Unable to get local issuer certificate&#8221; sounds like a client problem. It usually is not — it usually means your server is not sending the full chain. Learning to translate the message into the actual failure is most of the work.</p>



<h2 class="wp-block-heading">One command that tells you almost everything</h2>



<p class="wp-block-paragraph">Before touching config files, look at what the server is actually sending on the wire:</p>



<pre class="wp-block-code"><code>openssl s_client -connect example.com:443 -servername example.com -showcerts &lt;/dev/null</code></pre>



<p class="wp-block-paragraph">The <code>-servername</code> flag matters more than people expect. It sets SNI, and without it you may be handed the server&#8217;s default certificate rather than the one for the domain you care about. Plenty of confusing &#8220;the certificate is for the wrong domain&#8221; reports come from testing without SNI.</p>



<p class="wp-block-paragraph">Three parts of the output are worth reading carefully:</p>



<ul class="wp-block-list">
<li><strong>The &#8220;Certificate chain&#8221; block</strong> near the top, listing each certificate the server sent with its subject and issuer.</li>

<li><strong>The &#8220;Verify return code&#8221;</strong> at the bottom. <code>0 (ok)</code> is what you want. <code>20</code> and <code>21</code> both point at chain problems.</li>

<li><strong>The subject line of the first certificate</strong>, which tells you whether you even got the certificate you expected.</li>
</ul>



<p class="wp-block-paragraph">If you want the certificate details without the chain noise, pipe it into <code>x509</code>:</p>



<pre class="wp-block-code"><code>echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null 
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName</code></pre>



<p class="wp-block-paragraph">That single line answers &#8220;who issued it&#8221;, &#8220;when does it expire&#8221;, and &#8220;what hostnames does it cover&#8221; in one shot. It is the first thing I run.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family one: certificate chain issues</h2>



<p class="wp-block-paragraph">A publicly trusted certificate is almost never signed directly by a root. There is at least one intermediate CA in between. The client already trusts the root — it ships in the operating system or browser trust store — but it has no idea about the intermediate unless you hand it over. Your server&#8217;s job is to send the leaf certificate plus every intermediate needed to walk up to a trusted root.</p>



<h3 class="wp-block-heading">The classic mistake: serving the leaf without the intermediates</h3>



<p class="wp-block-paragraph">If you use Let&#8217;s Encrypt, certbot writes several files into <code>/etc/letsencrypt/live/example.com/</code>. Two of them look interchangeable and are not:</p>



<ul class="wp-block-list">
<li><code>cert.pem</code> — the leaf certificate on its own.</li>

<li><code>fullchain.pem</code> — the leaf plus the intermediates.</li>

<li><code>chain.pem</code> — the intermediates on their own.</li>

<li><code>privkey.pem</code> — the private key.</li>
</ul>



<p class="wp-block-paragraph">Nginx wants <code>fullchain.pem</code>. Point it at <code>cert.pem</code> and browsers will still work, so the mistake survives testing and surfaces weeks later when an API client fails.</p>



<pre class="wp-block-code"><code>server {
    listen 443 ssl;
    server_name example.com www.example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}</code></pre>



<p class="wp-block-paragraph">On Apache 2.4.8 and later, <code>SSLCertificateFile</code> can hold the leaf and the intermediates concatenated together, and <code>SSLCertificateChainFile</code> is deprecated. If you inherited a config that still uses the old directive, it will work, but it is worth cleaning up:</p>



<pre class="wp-block-code"><code>SSLCertificateFile    /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem</code></pre>



<h3 class="wp-block-heading">Wrong order, or a root that should not be there</h3>



<p class="wp-block-paragraph">Order is part of the specification, not a suggestion. The leaf comes first, then each issuer in sequence walking upward. Some clients tolerate a shuffled bundle; several do not, and the ones that do not tend to be the ones you find out about from an angry customer.</p>



<p class="wp-block-paragraph">Including the root certificate at the end is a different case: it is harmless from a trust perspective, since the client ignores it and uses its own copy, but it adds bytes to every single handshake for no benefit. Leave it out.</p>



<h3 class="wp-block-heading">Verifying a chain offline</h3>



<p class="wp-block-paragraph">You can check a bundle before deploying it, which is far better than discovering the problem in production:</p>



<pre class="wp-block-code"><code># Does the leaf verify against the intermediates you plan to serve?
openssl verify -untrusted chain.pem cert.pem

# Print the subject and issuer of every certificate in a bundle, in order
openssl crl2pkcs7 -nocrl -certfile fullchain.pem 
  | openssl pkcs7 -print_certs -noout</code></pre>



<p class="wp-block-paragraph">That second command is the one I reach for most. Read the output top to bottom: the issuer of each certificate should be the subject of the next one down. The moment that pattern breaks, you have found your gap.</p>



<h3 class="wp-block-heading">The key and the certificate do not match</h3>



<p class="wp-block-paragraph">Less common, but it produces a spectacularly unhelpful error. After a renewal where files were copied around by hand, confirm the pair still belongs together — the two hashes must be identical:</p>



<pre class="wp-block-code"><code>openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa  -noout -modulus -in privkey.pem | openssl md5</code></pre>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family two: SAN mismatches</h2>



<p class="wp-block-paragraph">The Common Name field is dead. It has been deprecated for hostname matching for years, and Chrome stopped honouring it entirely back in 2017. What matters now is the Subject Alternative Name extension. If the hostname a client requested is not in the SAN list, the connection fails, regardless of what the CN says.</p>



<p class="wp-block-paragraph">Check what a certificate actually covers:</p>



<pre class="wp-block-code"><code># From a file
openssl x509 -noout -ext subjectAltName -in cert.pem

# From a live server
echo | openssl s_client -connect example.com:443 -servername example.com 2&gt;/dev/null 
  | openssl x509 -noout -ext subjectAltName</code></pre>



<h3 class="wp-block-heading">The four SAN mistakes that keep recurring</h3>



<ul class="wp-block-list">
<li><strong>Apex without www, or www without apex.</strong> These are two distinct names. A certificate for <code>example.com</code> does not cover <code>www.example.com</code>. Both need to be in the SAN list, and both need to be passed to certbot with separate <code>-d</code> flags.</li>

<li><strong>Assuming a wildcard covers everything.</strong> It does not. <code>*.example.com</code> matches exactly one label. It covers <code>api.example.com</code>. It does not cover <code>example.com</code> itself, and it does not cover <code>staging.api.example.com</code>. Multi-level subdomains need their own wildcard or their own entry.</li>

<li><strong>A new subdomain that never made it into the renewal.</strong> You add <code>metrics.example.com</code> to nginx, point DNS at it, and forget that the certificate was issued for a fixed list of names. Adding a name means reissuing, not just reloading.</li>

<li><strong>Connecting by IP address.</strong> Almost no public certificate includes an IP SAN, so hitting the server by IP will always fail hostname verification. That is expected behaviour, not a broken certificate.</li>
</ul>



<p class="wp-block-paragraph">One more that catches people out: if you use Cloudflare in proxied mode, the certificate a visitor sees is Cloudflare&#8217;s edge certificate, not yours. Debugging the origin means bypassing the proxy and testing the origin IP directly with the <code>Host</code> header set correctly, or temporarily turning the orange cloud grey.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family three: expiry, and why automation is no longer optional</h2>



<p class="wp-block-paragraph">Expiry used to be a calendar problem. You bought a certificate for a year, set a reminder, and dealt with it once. That era is closing.</p>



<p class="wp-block-paragraph">In April 2025 the CA/Browser Forum passed ballot SC-081v3, which phases the maximum validity period for publicly trusted TLS certificates down from 398 days to 47 days. The schedule runs in three steps: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029. The first phase is already in force. Domain validation reuse windows shrink alongside it.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">The 47-day target was chosen deliberately: short enough that manual renewal stops being viable at any meaningful scale. Automation is the point of the change, not a side effect of it.</p>
</blockquote>



<p class="wp-block-paragraph">If you run a handful of domains, this is a nuisance. If you run fifty, manual renewal is already finished as a strategy.</p>



<h3 class="wp-block-heading">Getting ACME renewal right</h3>



<p class="wp-block-paragraph">Certbot installs a systemd timer or cron entry that attempts renewal twice a day and does nothing unless the certificate is close to expiry. Confirm it is actually there:</p>



<pre class="wp-block-code"><code>systemctl list-timers | grep certbot
certbot renew --dry-run</code></pre>



<p class="wp-block-paragraph">The dry run exercises the whole path against the staging environment without touching rate limits. Run it after any change to your web server config, not just after installing certbot. A renewal that worked last quarter can break because someone added a redirect that intercepts the ACME challenge path.</p>



<h3 class="wp-block-heading">The trap: renewal succeeds, the site still serves the old certificate</h3>



<p class="wp-block-paragraph">This is the failure I see most often, and it is genuinely maddening because every log says success. Certbot writes the new certificate to disk. Nginx has the old one loaded in memory and keeps serving it until it is told otherwise. The certificate on disk is valid, the certificate on the wire is expired, and everything looks fine from the server&#8217;s point of view.</p>



<p class="wp-block-paragraph">The fix is a deploy hook, which runs only when a certificate was actually renewed:</p>



<pre class="wp-block-code"><code>certbot renew --deploy-hook "systemctl reload nginx"</code></pre>



<p class="wp-block-paragraph">Better still, make it permanent by adding it to the renewal configuration at <code>/etc/letsencrypt/renewal/example.com.conf</code>, under the <code>[renewalparams]</code> section, so it survives however the renewal happens to be triggered:</p>



<pre class="wp-block-code"><code>renew_hook = systemctl reload nginx</code></pre>



<p class="wp-block-paragraph">Use <code>reload</code> rather than <code>restart</code>. A reload picks up the new certificate without dropping connections.</p>



<p class="wp-block-paragraph">Remember that anything else terminating TLS needs the same treatment. A mail server, a database proxy, a Docker container with the certificate baked into its image — each one needs its own reload step, and containers in particular will happily run for months with an expired certificate mounted from the host.</p>



<h3 class="wp-block-heading">Wildcards need DNS-01</h3>



<p class="wp-block-paragraph">Wildcard certificates cannot be issued through HTTP-01 validation. They require DNS-01, which means certbot needs to create a TXT record on your domain, which means API credentials for your DNS provider. Use a provider plugin rather than the manual mode — manual DNS-01 cannot be automated, and an unautomated wildcard on a 47-day cycle is an outage with a scheduled start time.</p>



<p class="wp-block-paragraph">If issuance fails outright with a policy error, check your CAA records before assuming the ACME client is broken. A CAA record that names a different CA will block issuance entirely:</p>



<pre class="wp-block-code"><code>dig CAA example.com +short</code></pre>



<h3 class="wp-block-heading">Monitor from outside, not from the server</h3>



<p class="wp-block-paragraph">Checking the certificate file on disk tells you what should be served. Only an external check tells you what is being served. Given the reload trap above, that difference is the entire point.</p>



<p class="wp-block-paragraph">A minimal check that works anywhere:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# Usage: ./cert-days.sh example.com
# Exits non-zero if fewer than 21 days remain.
set -euo pipefail

DOMAIN="$1"
THRESHOLD=21

END=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "${DOMAIN}" 2&gt;/dev/null 
      | openssl x509 -noout -enddate | cut -d= -f2)

DAYS=$(( ( $(date -d "${END}" +%s) - $(date +%s) ) / 86400 ))
echo "${DOMAIN}: ${DAYS} days remaining (expires ${END})"

[ "${DAYS}" -lt "${THRESHOLD}" ] &amp;&amp; exit 1
exit 0</code></pre>



<p class="wp-block-paragraph">If you already run Prometheus, the blackbox exporter gives you this for free through the <code>probe_ssl_earliest_cert_expiry</code> metric. An alert rule looks like this:</p>



<pre class="wp-block-code"><code>groups:
  - name: tls
    rules:
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 &lt; 21
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "TLS certificate expiring in under 21 days"</code></pre>



<p class="wp-block-paragraph">Set the threshold well above your renewal window. With 200-day certificates renewing at 30 days remaining, a 21-day alert gives you nine days of warning that something in the pipeline broke. Once 47-day certificates arrive, those numbers compress hard, and an alert that fires too late is the same as no alert.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A workflow for diagnosing TLS certificate errors</h2>



<p class="wp-block-paragraph">When something breaks, work through it in this order. It takes about a minute and rules out whole families of causes as you go.</p>



<ol class="wp-block-list">
<li><strong>Reproduce outside the browser.</strong> Use <code>curl -v</code> or <code>openssl s_client</code>. If the browser works and curl does not, you are almost certainly looking at a chain problem.</li>

<li><strong>Read the verify return code.</strong> Code 20 or 21 means a chain issue. A hostname mismatch message means a SAN issue. A date error means expiry — or a wrong system clock on the client.</li>

<li><strong>Print the dates.</strong> Rules expiry in or out in one command, and tells you whether the certificate on the wire matches the one on disk.</li>

<li><strong>Print the SANs.</strong> Confirm the exact hostname being requested appears in the list. Watch for apex versus www.</li>

<li><strong>Walk the chain.</strong> Each certificate&#8217;s issuer should be the next one&#8217;s subject. Find where the sequence breaks.</li>

<li><strong>Compare disk against wire.</strong> If the file is newer than what is being served, you have a reload problem, not a certificate problem.</li>

<li><strong>Check the client&#8217;s trust store.</strong> If everything above is clean and one specific client still fails, the problem has moved to their end — an old Android device, a Java service with its own <code>cacerts</code>, or a container image built on a base with a stale CA bundle.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes worth avoiding</h2>



<ul class="wp-block-list">
<li>Testing only in a browser, and mistaking its silent chain repair for a working configuration.</li>

<li>Pointing the web server at <code>cert.pem</code> instead of <code>fullchain.pem</code>.</li>

<li>Renewing without reloading the service that holds the certificate in memory.</li>

<li>Running <code>certbot renew</code> on a schedule but never running <code>--dry-run</code> after config changes.</li>

<li>Copying key and certificate files between servers by hand and losing track of which pair belongs together.</li>

<li>Monitoring the file on disk rather than the certificate actually being served.</li>

<li>Forgetting that mail servers, load balancers, and containers terminate TLS too, and need their own renewal path.</li>

<li>Adding a subdomain to the web server config and assuming the existing certificate covers it.</li>
</ul>



<h2 class="wp-block-heading">Best practices that hold up over time</h2>



<ul class="wp-block-list">
<li>Automate issuance and renewal from day one, even for a single domain. The habit matters more than the scale.</li>

<li>Always use a deploy hook, and always reload rather than restart.</li>

<li>Alert on days remaining, measured externally, with a threshold at least twice your renewal window.</li>

<li>Keep an inventory of every hostname you hold a certificate for. You cannot monitor what you have forgotten about.</li>

<li>Prefer DNS-01 with a provider plugin if you need wildcards, and never rely on manual DNS validation.</li>

<li>Set CAA records deliberately so you know which CAs are permitted to issue for your domain.</li>

<li>Test with a non-browser client before calling a certificate deployment finished.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why does my site work in Chrome but fail in curl?</h3>



<p class="wp-block-paragraph">Almost always an incomplete chain. Browsers fetch missing intermediate certificates automatically using the Authority Information Access extension; curl does not. Point your web server at the full chain file rather than the leaf certificate and both will work.</p>



<h3 class="wp-block-heading">What does &#8220;unable to get local issuer certificate&#8221; actually mean?</h3>



<p class="wp-block-paragraph">The client could not build a path from your certificate to a root it trusts. Nine times out of ten the server is not sending its intermediates. Occasionally the client&#8217;s own CA bundle is out of date, which is common in older container base images.</p>



<h3 class="wp-block-heading">Does a wildcard certificate cover the root domain?</h3>



<p class="wp-block-paragraph">No. <code>*.example.com</code> matches a single label, so it covers <code>www.example.com</code> but not <code>example.com</code> and not <code>a.b.example.com</code>. Include the apex explicitly as an additional SAN entry when you request the certificate.</p>



<h3 class="wp-block-heading">My certificate renewed but the site still shows it as expired. Why?</h3>



<p class="wp-block-paragraph">The service holding the certificate has not reloaded. The new file is on disk; the old one is still in memory. Add a deploy hook that reloads the service on renewal, then verify from outside with <code>openssl s_client</code> rather than trusting the file timestamp.</p>



<h3 class="wp-block-heading">How short will certificate lifetimes actually get?</h3>



<p class="wp-block-paragraph">Under the CA/Browser Forum schedule, 200 days applies from March 2026, 100 days from March 2027, and 47 days from March 2029. Private and internal CAs are not bound by these rules, but public certificates are.</p>



<h3 class="wp-block-heading">Is the Common Name field still used at all?</h3>



<p class="wp-block-paragraph">Not for hostname validation. Modern clients read only the Subject Alternative Name extension. A certificate whose CN matches but whose SAN list does not include the hostname will be rejected.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">TLS certificate errors feel opaque mostly because the error text points at the wrong layer. Once you sort them into chain, SAN, or expiry, each one has a short diagnostic path and an obvious fix. Get comfortable with <code>openssl s_client</code> and <code>openssl x509</code> and you will resolve most incidents faster than it takes to find the right dashboard.</p>



<p class="wp-block-paragraph">The expiry category is the one worth engineering away permanently. With maximum lifetimes dropping toward 47 days, any process that depends on a person remembering something will fail eventually. Automate the renewal, add the deploy hook, and alert on what is actually being served rather than what is sitting on disk. Do that once and TLS certificate errors go back to being a rare curiosity instead of a recurring outage.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with TLS, certificates, or infrastructure automation?</h2>



<p class="wp-block-paragraph">I work with teams on exactly this kind of problem — broken certificate chains, ACME automation across multiple servers, expiry monitoring wired into Prometheus and Grafana, and the wider DevOps plumbing that keeps it all running quietly in the background.</p>



<p class="wp-block-paragraph">If you would rather hand this off than debug it at two in the morning, you can hire me on Upwork.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/tls-certificate-errors/">TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
