That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.

So this post compares Nginx vs Apache vs Caddy on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.

Why the benchmark comparison picks the wrong winner

Almost every throughput comparison you’ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you’re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.

The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn’t most projects.

For everyone else, the useful question is not “which is fastest” but “which chores am I choosing to keep.” That’s what the rest of this is about.

Failure family one: the process model decides how you fall over

This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.

Apache: a hard worker ceiling, and a queue behind it

Apache’s behaviour depends entirely on which Multi-Processing Module is loaded. With prefork, you get one process per connection, capped by MaxRequestWorkers. When you hit that cap, new connections don’t get rejected. They sit in the kernel’s listen backlog waiting for a worker to free up.

The symptom is the worst kind: the site “hangs.” Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.

Check which MPM you’re actually running before you theorise about anything:

# RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"

The reason prefork survives is mod_php. It isn’t safe with Apache’s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over mod_proxy_fcgi instead, which frees you to run mpm_event and gets you HTTP/2 as a side effect, since mod_http2 disables itself under prefork.

If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.

Nginx: cheap connections, different limits

Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.

Its ceiling is worker_processes multiplied by worker_connections, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That’s a genuine operational advantage: the failure announces itself.

The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.

Caddy: goroutines, and a softer edge

Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don’t tune a worker count at all.

The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean “workers exhausted” line, you get memory growth and garbage collection pressure. There’s no knob you turned wrong, which is pleasant right up until you need to explain what happened.

Failure family two: who owns the certificate lifecycle

This is the one that actually causes outages, and it is the sharpest genuine difference between the three.

With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.

The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:

# Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"

The distinction that matters: --deploy-hook runs only when a certificate was actually renewed, so you’re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.

Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.

It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy’s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an ask endpoint you host:

{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}

Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older interval and burst rate-limiting options are no longer recommended and should be removed if you still have them in a config.

One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.

Failure family three: where the configuration lives

This is the axis people underweight, and it’s usually the one that actually decides the answer.

Apache supports .htaccess: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.

The costs are real and they’re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for .htaccess files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.

Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with try_files rather than per-directory rule files. If you own the box, that’s a feature: nothing changes routing behind your back and the running configuration is exactly what’s in version control. If you host other people’s applications, it’s a support burden, because every customer rewrite rule becomes a ticket for you.

Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.

Nginx: where it wins and where it doesn’t

Where it wins: high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you’ll meet has a known-good Nginx config, and everyone you hire has already read one.

Where it doesn’t: the DNS behaviour catches people constantly. A hostname written directly into proxy_pass, or into an upstream block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says “stale DNS”; you just get connection timeouts to an address that looks plausible.

Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that’s what moves the lookup from config parse time to request time:

location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}

Note the $request_uri on the end. Once a variable appears in proxy_pass, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream’s root. Also worth knowing: an upstream block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.

The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But “who decides what ships” is a fair question to ask about any dependency, and this one now has a public answer.

Apache: where it wins and where it doesn’t

Where it wins: anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.

Where it doesn’t: memory per connection, if you’re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that’s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.

I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and mod_php to mpm_event and PHP-FPM first, then measure, and quite often stop there.

Caddy: where it wins and where it doesn’t

Where it wins: anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.

app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}

That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.

Where it doesn’t: the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with xcaddy rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there’s a subtler cost: automation you don’t understand is still automation you’re responsible for. Being paged about ACME behaviour you’ve never had to learn is a worse afternoon than being paged about Certbot.

On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the dynamic a upstream module with a refresh interval is the built-in answer, which is a cleaner story than Nginx’s variable trick.

A decision procedure for Nginx vs Apache vs Caddy

Work down this list and stop at the first one that matches. It’ll be right more often than a benchmark.

1. Do applications you don’t control need to change server behaviour? If yes, Apache. Nothing else has .htaccess, and reimplementing it in tickets is not a plan.
2. Is TLS renewal your top operational risk? Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.
3. Are you terminating a very high connection rate, or serving heavy static content? Nginx. This is the workload it was designed around.
4. Does the team already run one of these well? Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.
5. Is there a working Apache install and no specific complaint? Fix the MPM and PHP handler, then reassess. Most “we need Nginx” tickets are actually “we need to stop using prefork.”
6. None of the above? Nginx for anything that will be handed to another engineer. Caddy for anything you’ll run yourself.

Common mistakes

• Renewing certificates without reloading. The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.
• Migrating for benchmark numbers. You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.
• Leaving Apache on prefork after moving to PHP-FPM. The migration is only half done, and you’re still paying prefork’s memory cost while also giving up HTTP/2.
• Hostnames baked into proxy_pass with dynamic upstreams. Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.
• Enabling Caddy’s on-demand TLS without an ask endpoint. You’ve handed strangers the ability to make you request certificates.
• Exposing Caddy’s admin API beyond loopback. Unauthenticated full config replacement is exactly as bad as it sounds.
• Monitoring the certificate file instead of the handshake. The file is almost never the thing that’s wrong. Check what the server actually presents, from outside the box.

How I’d decide, in practice

• Pick based on who has to change the config six months from now, not on requests per second today.
• Whichever you choose, dump the effective configuration rather than reading files: nginx -T, apachectl -S, caddy adapt --config /etc/caddy/Caddyfile. Include files hide surprises.
• Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.
• Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.
• Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.
• Validate before reloading, every time. nginx -t, apachectl configtest and caddy validate all exist so you don’t find out from users.

Frequently asked questions

Is Caddy production ready?

Yes. It’s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.

Can I run Nginx and Apache together?

Yes, and it’s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their .htaccess behaviour. You get Nginx’s connection handling and Apache’s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.

Which is best for WordPress?

Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into try_files and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.

Does Nginx really re-resolve DNS for upstreams?

Not by default. A literal hostname in proxy_pass or an upstream block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a resolver directive, and remembering that this changes how the request URI is passed.

Should I switch from mod_php to PHP-FPM?

In almost all cases yes. It’s the change that unlocks Apache’s threaded MPMs and HTTP/2, and it separates PHP’s lifecycle from the web server’s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or .htaccess, since those move to a .user.ini file with different syntax.

Does the freenginx fork mean I should stop using Nginx?

No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn’t put a client on a fork without a specific reason.

Which uses the least memory?

Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on mpm_event, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.

The one thing worth remembering

The Nginx vs Apache vs Caddy question is not a performance question. All three are fast enough for what you’re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.

Pick the set of chores your team can actually carry. Then go fix the thing that’s genuinely slow, which is almost certainly not the web server.

Need a second opinion on your web server setup?

Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:

• Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does
• Migrating Apache from prefork and mod_php to mpm_event with PHP-FPM, without breaking existing .htaccess behaviour
• Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches
• Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring
• Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted
• Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren’t

Send me your config, an error log excerpt, or the output of nginx -T and tell me what you’re seeing. I’d rather look at the real thing than guess from a description.

The post Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects appeared first on John Nessime.

Your server is only sending the leaf certificate. The intermediate that links it to a trusted root is missing. Your browser papers over that: it either has the intermediate cached from another site, or it quietly fetches it using the AIA extension in the certificate. curl, Java, Go, Python and every other non-browser client do no such thing. They see a certificate signed by something they do not trust and stop.

This is the most common of all TLS certificate errors and it is close to invisible from a desktop. The padlock in your browser is not a test. It is a client with a large safety net doing you a favour.

This post covers the four things a client actually checks and how each one fails: the chain, the names, the clock, and the trust store. Then the automation half, which has its own quiet failure: a renewal that succeeds and never reaches the running server. Plus the two things about TLS that have genuinely changed recently and will break config you copied from an older guide.

Look at what the server is actually sending

Before theorising, get the facts. One command tells you almost everything.

# -servername sends SNI. Without it you get whatever the default
# vhost serves, which is a different certificate on most servers
# and the source of a lot of confusing results.
# -showcerts prints every certificate the server sent, in order.
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

Read the top of that output. There is a Certificate chain block listing what arrived, numbered from zero, with s: for subject and i: for issuer. Entry 0 is your certificate. Entry 1 should be the intermediate that issued it, and its subject should exactly match entry 0’s issuer.

If the chain block has only entry 0, that is your bug. Right at the bottom, Verify return code gives the verdict: 0 (ok) is what you want, 21 (unable to verify the first certificate) means the chain is incomplete, and 20 (unable to get local issuer certificate) means the issuer is not in the local trust store.

Run it from a machine that is not yours. A clean container is ideal, because your laptop has accumulated intermediates and corporate roots that a customer’s server has not.

Chain problems

Almost all of these come down to one file choice. ACME clients write out several files and two of them look interchangeable:

• cert.pem is the leaf alone. Using this is the mistake.
• chain.pem is the intermediates alone.
• fullchain.pem is leaf plus intermediates, in the right order. This is the one your server should point at.

# nginx
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

# Apache. On current versions the chain goes in SSLCertificateFile.
# SSLCertificateChainFile is deprecated; guides that still use it
# are old enough that other advice in them is probably stale too.
SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem

Order matters as well as contents. The leaf must come first, then each intermediate that signs the one above it. A chain assembled in the wrong order will be rejected by strict clients even though every certificate in it is correct. You can check a chain offline before deploying it:

# Does this leaf chain to a trusted root using these intermediates?
# Exits non-zero and explains itself if not.
openssl verify -untrusted chain.pem cert.pem

Do not include the root certificate in the file you serve. The client either has it or does not, and sending it wastes bytes on every handshake without changing any outcome.

Name problems

The second family. A certificate can be perfectly valid and still be the wrong certificate for the name being requested.

# The hostname the client asked for must appear in here.
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"

Three things people get wrong here:

• The Common Name does not count. Modern clients ignore the CN field entirely for hostname matching. If the name is only in the CN and not in the SAN list, it will fail, and the error message will not explain that clearly.
• Wildcards are exactly one label deep. *.example.com covers www.example.com and api.example.com. It does not cover example.com itself, and it does not cover a.b.example.com. The bare domain has to be listed separately, and it usually is not.
• SNI decides which certificate you get. One IP serving many sites picks the certificate based on the name the client sends during the handshake. A client that does not send SNI, or an openssl command without -servername, gets the default vhost’s certificate and produces a name mismatch that has nothing to do with your actual configuration.

Time and trust problems

# Local file
openssl x509 -in cert.pem -noout -dates

# Live endpoint, subject and dates in one go
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null 
  | openssl x509 -noout -dates -subject

Expired is obvious. notBefore in the future is the interesting one, because it almost never means the certificate is wrong. It means the client’s clock is wrong. Containers with no NTP, a VM restored from a snapshot, an embedded device with a dead battery: all of them produce “certificate not yet valid” against a perfectly good certificate. Check the clock on the failing machine before you touch anything on the server.

Trust store problems look similar but come from a different place. The root that signed your chain has to be in the trust store of the machine making the request, and old systems have old trust stores. A device that has not received an OS update in years may simply not know about a certificate authority that appeared since. There is no server-side fix for that beyond choosing a CA with broader legacy coverage, which is one of the few genuine arguments for a paid certificate.

The renewal that succeeded and changed nothing

Here is the second invisible failure, and it catches experienced people. Your renewal timer runs, the ACME client obtains a new certificate, writes it to disk, logs success. The site goes down anyway.

Nginx read the certificate into memory when it started. It does not watch the file. Neither does Apache, HAProxy, Postfix, or almost anything else. Until the service is reloaded it keeps serving the certificate it loaded at boot, which is the one that just expired. Everything in your logs says the renewal worked, because it did. The delivery step was never wired up.

Certbot handles this with deploy hooks, which run only when a certificate was actually renewed. Put a script in the hooks directory and it applies to every certificate on the box:

#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-services.sh
# Remember: chmod +x, or it silently never runs.
set -euo pipefail

# Validate the config BEFORE reloading. A reload with a broken
# config can take the site down at the exact moment you were
# trying to keep it up.
nginx -t
systemctl reload nginx

# Anything else that holds a copy of the certificate.
# systemctl reload postfix

Then prove the whole path works without waiting sixty days to find out:

# Exercises the full renewal against the staging environment.
# Installs nothing, counts against no rate limits, and will
# fail loudly if your challenge setup is broken.
certbot renew --dry-run

One caveat worth knowing: --dry-run does not run deploy hooks by default, precisely because nothing was deployed. So it validates the issuance path but not the reload. Test that separately by touching the service and confirming the served certificate changes.

Expiry automation, and why it is no longer optional

This part has a date attached, so it is worth stating plainly rather than dancing around it.

The CA/Browser Forum approved a phased reduction of maximum certificate lifetimes. The cap dropped from 398 days to 200 days on 15 March 2026, falls to 100 days on 15 March 2027, and reaches 47 days on 15 March 2029. Domain validation reuse periods shrink alongside it.

The practical consequence: any process that involves a person remembering to renew something once a year is already broken, and by the third phase you are renewing every six weeks. Whatever you were getting away with manually, stop. ACME with a timer is the baseline. Certbot is the default choice, acme.sh is lighter and easier to script around, and Caddy obtains and renews certificates with no configuration at all, which is a genuinely good reason to consider it for a simple reverse proxy.

Choosing a challenge type

• HTTP-01 proves control by serving a file over port 80. Simple, and it breaks whenever port 80 is closed, redirected wholesale to HTTPS in a way that mangles the challenge path, or sitting behind a proxy that does not forward it.
• DNS-01 proves control by writing a TXT record. More setup, needs an API token for your DNS provider, and it is the only option for wildcard certificates. It also works for hosts with no inbound internet access at all, which is why I reach for it on anything internal.

The trade-off with DNS-01 is a credential: your renewal process now holds a token that can edit your DNS zone. Scope it to the narrowest permission your provider offers, and if they support delegating just the _acme-challenge records to a separate zone, do that instead.

Monitor expiry externally

Do not rely on the renewal system reporting its own health. If it is broken, it is also probably not telling you. Let’s Encrypt has discontinued its expiration reminder emails, so if that was quietly acting as your safety net, it is not one any more.

Check from outside, against the live endpoint, the way a user would:

#!/usr/bin/env bash
set -euo pipefail

HOST="example.com"
DAYS=21

# -checkend takes a number of seconds and exits non-zero if the
# certificate expires within that window. No parsing dates by hand.
if ! openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null 
     | openssl x509 -noout -checkend $(( DAYS * 86400 )); then
  echo "TLS certificate for $HOST expires within $DAYS days" >&2
  exit 1
fi

If you already run Prometheus, the blackbox exporter gives you probe_ssl_earliest_cert_expiry for free, which is a timestamp you can alert on and graph in Grafana across every endpoint you own. That is the better answer at more than a handful of hosts, because it also catches the certificates nobody remembered existed.

Two pieces of config that are now wrong

OCSP stapling with Let’s Encrypt. Let’s Encrypt removed the OCSP responder URL from newly issued certificates in May 2025 and shut down its responders in August 2025. If your config still has this, it does nothing except print a warning on every reload:

# nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate
# Comment these out if your certificates come from Let's Encrypt.
# ssl_stapling on;
# ssl_stapling_verify on;

Nothing breaks by leaving them, but a warning that appears on every reload is a warning people learn to ignore, and then they ignore the next one too.

Hand-tuned cipher suite lists. If you pasted a ssl_ciphers line from a guide more than a couple of years old, re-check it against the current Mozilla SSL Configuration Generator. Lists that were considered hardened at the time now sometimes exclude things you want or include things you do not, and the defaults have improved considerably.

When there is a CDN in front

Behind Cloudflare or a similar edge, there are two separate TLS connections: browser to edge, and edge to your origin. They use different certificates and fail independently.

This creates a specific blind spot. Your origin certificate can be expired, self-signed or serving the wrong name, and visitors will see nothing wrong, because the edge presents its own valid certificate and, in the more permissive encryption modes, does not verify the origin. It works until you switch to strict origin validation, or until the edge is bypassed, and then everything fails at once for reasons that look unrelated to anything you changed.

Test the origin directly by name, bypassing DNS. If your origin is a VPS with a provider like InterServer and the public DNS points at the CDN, this is the only way to see what the origin is really serving:

# --resolve forces the hostname to a specific IP without touching
# /etc/hosts, so SNI and Host header stay correct.
curl -v --resolve example.com:443:203.0.113.10 https://example.com/

Troubleshooting by error message

unable to get local issuer certificate

Missing intermediate on the server, or a trust store on the client that lacks the root. Check the server first with -showcerts; if the chain is complete, the problem is on the client side.

certificate has expired, but I just renewed it

The service was never reloaded. Compare the dates on disk with the dates being served. If they differ, that is your answer, and the fix is a deploy hook rather than another manual renewal.

Hostname mismatch on a name that is definitely in the certificate

Either SNI is not reaching the right vhost, or the name is in the Common Name rather than the SAN list, or you are hitting a wildcard at the wrong depth. Check the SAN list and re-run s_client with an explicit -servername.

Works over IPv4, fails over IPv6 (or the reverse)

Two different machines are answering, and only one of them was updated. Force the family with curl -4 and curl -6 and compare. Load balancers with one member out of sync produce the same intermittent pattern.

Renewal fails with a challenge error

For HTTP-01, confirm that http://example.com/.well-known/acme-challenge/test is reachable from outside without being redirected or rewritten. A blanket redirect to HTTPS is the usual culprit. For DNS-01, check that the TXT record actually appears with dig, and give slow-propagating providers more time.

Everything looks right and it still fails

Run an external scan. SSL Labs or a local run of testssl.sh will check things you did not think to, including protocol support and chain issues from the perspective of clients you do not have installed.

Common mistakes

• Pointing the server at cert.pem instead of fullchain.pem.
• Treating a padlock in your own browser as proof the certificate is correctly deployed.
• Testing with openssl s_client without -servername and debugging the wrong certificate.
• Assuming *.example.com covers the bare domain.
• Putting the hostname only in the Common Name.
• Renewing without reloading the service.
• Forgetting chmod +x on a deploy hook, so it silently never runs.
• Relying on the renewal system to tell you when it is broken.
• Including the root certificate in the served chain.
• Leaving OCSP stapling enabled with certificates that no longer support it.
• Ignoring the origin certificate because the CDN hides its state.
• Debugging a “not yet valid” certificate on the server when the client’s clock is wrong.

Best practices

• Serve the full chain, leaf first, without the root.
• Verify from a clean machine or container, never from the laptop you browse on.
• Put every name you serve in the SAN list, including the bare domain alongside any wildcard.
• Automate renewal with ACME and a timer, and run certbot renew --dry-run after any change to the setup.
• Attach a deploy hook that validates the config before reloading, and make it executable.
• Monitor expiry externally against the live endpoint, and alert with weeks of headroom.
• Use DNS-01 for wildcards and internal hosts, with a narrowly scoped API token.
• Keep NTP running everywhere, including inside containers.
• Test the origin directly when there is a CDN in front, not just the public hostname.
• Inventory your certificates, because the one that takes you down is the one nobody remembered.

FAQ

Why does my certificate work in Chrome but fail in curl?

Your server is not sending the intermediate certificate. Browsers work around this by caching intermediates from other sites and by fetching them from the URL in the certificate’s AIA extension. Command-line tools and application HTTP libraries do neither. Switch the server to fullchain.pem and reload.

What is the difference between cert.pem, chain.pem and fullchain.pem?

Leaf only, intermediates only, and both concatenated in the correct order. Web servers want fullchain.pem. The private key stays in privkey.pem and never goes in any of them.

Does a wildcard certificate cover the root domain?

No. *.example.com matches exactly one label, so it covers www.example.com but not example.com and not a.b.example.com. Request the bare domain as an additional SAN, which most ACME clients will do if you pass both names.

How often should certificates renew now?

Whatever your CA issues, renew at roughly two-thirds of the lifetime so a failed attempt has room to retry before anything expires. Since maximum lifetimes are on a published downward schedule, build for automated renewal at any interval rather than tuning a number you will have to change again.

Should I still enable OCSP stapling?

Not with Let’s Encrypt certificates, which no longer carry a responder URL. If your CA still supports it, stapling remains worthwhile. Check whether your certificate actually contains an OCSP URL before enabling the directives, rather than assuming.

Is a paid certificate more secure than a free one?

Cryptographically, no. The encryption is identical. What money buys is broader compatibility with old trust stores, longer validation reuse, warranties, and human support when something goes wrong. For a normal public website, a free automated certificate is the right default.

How do I get a certificate for something with no public DNS or open ports?

DNS-01, on a public domain you control, pointed at whatever internal address you like. The CA only needs to see the TXT record; it never needs to reach the host. This is how internal services get publicly trusted certificates without being exposed.

The one thing to remember

Most TLS certificate errors are not about cryptography. They are about delivery: whether the server sends everything the client needs, whether the name it sends matches the name that was asked for, and whether the certificate sitting in memory is the same one sitting on disk.

So stop checking with your browser. Check with openssl s_client from a machine that owes you no favours, and make sure something outside your server is watching the expiry date. As lifetimes keep shrinking, the gap between a renewal process that works and one that merely appears to work gets narrower every year.

Certificates causing you trouble?

Certificate problems tend to arrive at bad moments and turn out to be one wrong file path. Things I help with:

• Diagnosing a certificate that works for some clients and not others, and telling you exactly which link in the chain is missing.
• Setting up ACME renewal with deploy hooks across nginx, Apache, HAProxy, mail servers and anything else holding a copy.
• Migrating to DNS-01 for wildcards and internal hosts, with properly scoped API credentials.
• Certificate inventory and expiry monitoring in Prometheus and Grafana, so nothing expires without warning.
• Preparing a certificate estate for shorter lifetimes, replacing anything that still depends on a person remembering.
• Origin and edge TLS configuration behind a CDN, including strict origin validation.

Send me the output of openssl s_client -connect yourhost:443 -servername yourhost -showcerts and the error your failing client reports, and I will tell you what is wrong.

The post It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry appeared first on John Nessime.

Nearly every TLS certificate error I have had to chase down comes back to one of three things: the chain your server sends is incomplete, the certificate does not actually cover the hostname being requested, or it expired and no alert fired. This article walks through all three — how to identify which one you are looking at, the exact commands to prove it, and how to set up renewal automation so the third category stops being a category at all.

Why TLS certificate errors are so confusing to debug

Two things conspire to make this harder than it should be.

The first is that browsers are forgiving in ways other clients are not. When a server sends an incomplete certificate chain, most modern browsers will quietly fetch the missing intermediate using the Authority Information Access extension embedded in the certificate. They fix your misconfiguration on the fly and show you a padlock. Meanwhile curl, Python’s requests, Java clients, Go services, and older Android devices do no such thing. They fail. This is why “it works in Chrome” is worth almost nothing as a diagnostic signal.

The second is that error messages describe the symptom rather than the cause. “Unable to get local issuer certificate” sounds like a client problem. It usually is not — it usually means your server is not sending the full chain. Learning to translate the message into the actual failure is most of the work.

One command that tells you almost everything

Before touching config files, look at what the server is actually sending on the wire:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

The -servername flag matters more than people expect. It sets SNI, and without it you may be handed the server’s default certificate rather than the one for the domain you care about. Plenty of confusing “the certificate is for the wrong domain” reports come from testing without SNI.

Three parts of the output are worth reading carefully:

• The “Certificate chain” block near the top, listing each certificate the server sent with its subject and issuer.
• The “Verify return code” at the bottom. 0 (ok) is what you want. 20 and 21 both point at chain problems.
• The subject line of the first certificate, which tells you whether you even got the certificate you expected.

If you want the certificate details without the chain noise, pipe it into x509:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null 
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

That single line answers “who issued it”, “when does it expire”, and “what hostnames does it cover” in one shot. It is the first thing I run.

Family one: certificate chain issues

A publicly trusted certificate is almost never signed directly by a root. There is at least one intermediate CA in between. The client already trusts the root — it ships in the operating system or browser trust store — but it has no idea about the intermediate unless you hand it over. Your server’s job is to send the leaf certificate plus every intermediate needed to walk up to a trusted root.

The classic mistake: serving the leaf without the intermediates

If you use Let’s Encrypt, certbot writes several files into /etc/letsencrypt/live/example.com/. Two of them look interchangeable and are not:

• cert.pem — the leaf certificate on its own.
• fullchain.pem — the leaf plus the intermediates.
• chain.pem — the intermediates on their own.
• privkey.pem — the private key.

Nginx wants fullchain.pem. Point it at cert.pem and browsers will still work, so the mistake survives testing and surfaces weeks later when an API client fails.

server {
    listen 443 ssl;
    server_name example.com www.example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}

On Apache 2.4.8 and later, SSLCertificateFile can hold the leaf and the intermediates concatenated together, and SSLCertificateChainFile is deprecated. If you inherited a config that still uses the old directive, it will work, but it is worth cleaning up:

SSLCertificateFile    /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem

Wrong order, or a root that should not be there

Order is part of the specification, not a suggestion. The leaf comes first, then each issuer in sequence walking upward. Some clients tolerate a shuffled bundle; several do not, and the ones that do not tend to be the ones you find out about from an angry customer.

Including the root certificate at the end is a different case: it is harmless from a trust perspective, since the client ignores it and uses its own copy, but it adds bytes to every single handshake for no benefit. Leave it out.

Verifying a chain offline

You can check a bundle before deploying it, which is far better than discovering the problem in production:

# Does the leaf verify against the intermediates you plan to serve?
openssl verify -untrusted chain.pem cert.pem

# Print the subject and issuer of every certificate in a bundle, in order
openssl crl2pkcs7 -nocrl -certfile fullchain.pem 
  | openssl pkcs7 -print_certs -noout

That second command is the one I reach for most. Read the output top to bottom: the issuer of each certificate should be the subject of the next one down. The moment that pattern breaks, you have found your gap.

The key and the certificate do not match

Less common, but it produces a spectacularly unhelpful error. After a renewal where files were copied around by hand, confirm the pair still belongs together — the two hashes must be identical:

openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa  -noout -modulus -in privkey.pem | openssl md5

Family two: SAN mismatches

The Common Name field is dead. It has been deprecated for hostname matching for years, and Chrome stopped honouring it entirely back in 2017. What matters now is the Subject Alternative Name extension. If the hostname a client requested is not in the SAN list, the connection fails, regardless of what the CN says.

Check what a certificate actually covers:

# From a file
openssl x509 -noout -ext subjectAltName -in cert.pem

# From a live server
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null 
  | openssl x509 -noout -ext subjectAltName

The four SAN mistakes that keep recurring

• Apex without www, or www without apex. These are two distinct names. A certificate for example.com does not cover www.example.com. Both need to be in the SAN list, and both need to be passed to certbot with separate -d flags.
• Assuming a wildcard covers everything. It does not. *.example.com matches exactly one label. It covers api.example.com. It does not cover example.com itself, and it does not cover staging.api.example.com. Multi-level subdomains need their own wildcard or their own entry.
• A new subdomain that never made it into the renewal. You add metrics.example.com to nginx, point DNS at it, and forget that the certificate was issued for a fixed list of names. Adding a name means reissuing, not just reloading.
• Connecting by IP address. Almost no public certificate includes an IP SAN, so hitting the server by IP will always fail hostname verification. That is expected behaviour, not a broken certificate.

One more that catches people out: if you use Cloudflare in proxied mode, the certificate a visitor sees is Cloudflare’s edge certificate, not yours. Debugging the origin means bypassing the proxy and testing the origin IP directly with the Host header set correctly, or temporarily turning the orange cloud grey.

Family three: expiry, and why automation is no longer optional

Expiry used to be a calendar problem. You bought a certificate for a year, set a reminder, and dealt with it once. That era is closing.

In April 2025 the CA/Browser Forum passed ballot SC-081v3, which phases the maximum validity period for publicly trusted TLS certificates down from 398 days to 47 days. The schedule runs in three steps: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029. The first phase is already in force. Domain validation reuse windows shrink alongside it.

The 47-day target was chosen deliberately: short enough that manual renewal stops being viable at any meaningful scale. Automation is the point of the change, not a side effect of it.

If you run a handful of domains, this is a nuisance. If you run fifty, manual renewal is already finished as a strategy.

Getting ACME renewal right

Certbot installs a systemd timer or cron entry that attempts renewal twice a day and does nothing unless the certificate is close to expiry. Confirm it is actually there:

systemctl list-timers | grep certbot
certbot renew --dry-run

The dry run exercises the whole path against the staging environment without touching rate limits. Run it after any change to your web server config, not just after installing certbot. A renewal that worked last quarter can break because someone added a redirect that intercepts the ACME challenge path.

The trap: renewal succeeds, the site still serves the old certificate

This is the failure I see most often, and it is genuinely maddening because every log says success. Certbot writes the new certificate to disk. Nginx has the old one loaded in memory and keeps serving it until it is told otherwise. The certificate on disk is valid, the certificate on the wire is expired, and everything looks fine from the server’s point of view.

The fix is a deploy hook, which runs only when a certificate was actually renewed:

certbot renew --deploy-hook "systemctl reload nginx"

Better still, make it permanent by adding it to the renewal configuration at /etc/letsencrypt/renewal/example.com.conf, under the [renewalparams] section, so it survives however the renewal happens to be triggered:

renew_hook = systemctl reload nginx

Use reload rather than restart. A reload picks up the new certificate without dropping connections.

Remember that anything else terminating TLS needs the same treatment. A mail server, a database proxy, a Docker container with the certificate baked into its image — each one needs its own reload step, and containers in particular will happily run for months with an expired certificate mounted from the host.

Wildcards need DNS-01

Wildcard certificates cannot be issued through HTTP-01 validation. They require DNS-01, which means certbot needs to create a TXT record on your domain, which means API credentials for your DNS provider. Use a provider plugin rather than the manual mode — manual DNS-01 cannot be automated, and an unautomated wildcard on a 47-day cycle is an outage with a scheduled start time.

If issuance fails outright with a policy error, check your CAA records before assuming the ACME client is broken. A CAA record that names a different CA will block issuance entirely:

dig CAA example.com +short

Monitor from outside, not from the server

Checking the certificate file on disk tells you what should be served. Only an external check tells you what is being served. Given the reload trap above, that difference is the entire point.

A minimal check that works anywhere:

#!/usr/bin/env bash
# Usage: ./cert-days.sh example.com
# Exits non-zero if fewer than 21 days remain.
set -euo pipefail

DOMAIN="$1"
THRESHOLD=21

END=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "${DOMAIN}" 2>/dev/null 
      | openssl x509 -noout -enddate | cut -d= -f2)

DAYS=$(( ( $(date -d "${END}" +%s) - $(date +%s) ) / 86400 ))
echo "${DOMAIN}: ${DAYS} days remaining (expires ${END})"

[ "${DAYS}" -lt "${THRESHOLD}" ] && exit 1
exit 0

If you already run Prometheus, the blackbox exporter gives you this for free through the probe_ssl_earliest_cert_expiry metric. An alert rule looks like this:

groups:
  - name: tls
    rules:
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 21
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "TLS certificate expiring in under 21 days"

Set the threshold well above your renewal window. With 200-day certificates renewing at 30 days remaining, a 21-day alert gives you nine days of warning that something in the pipeline broke. Once 47-day certificates arrive, those numbers compress hard, and an alert that fires too late is the same as no alert.

A workflow for diagnosing TLS certificate errors

When something breaks, work through it in this order. It takes about a minute and rules out whole families of causes as you go.

1. Reproduce outside the browser. Use curl -v or openssl s_client. If the browser works and curl does not, you are almost certainly looking at a chain problem.
2. Read the verify return code. Code 20 or 21 means a chain issue. A hostname mismatch message means a SAN issue. A date error means expiry — or a wrong system clock on the client.
3. Print the dates. Rules expiry in or out in one command, and tells you whether the certificate on the wire matches the one on disk.
4. Print the SANs. Confirm the exact hostname being requested appears in the list. Watch for apex versus www.
5. Walk the chain. Each certificate’s issuer should be the next one’s subject. Find where the sequence breaks.
6. Compare disk against wire. If the file is newer than what is being served, you have a reload problem, not a certificate problem.
7. Check the client’s trust store. If everything above is clean and one specific client still fails, the problem has moved to their end — an old Android device, a Java service with its own cacerts, or a container image built on a base with a stale CA bundle.

Common mistakes worth avoiding

• Testing only in a browser, and mistaking its silent chain repair for a working configuration.
• Pointing the web server at cert.pem instead of fullchain.pem.
• Renewing without reloading the service that holds the certificate in memory.
• Running certbot renew on a schedule but never running --dry-run after config changes.
• Copying key and certificate files between servers by hand and losing track of which pair belongs together.
• Monitoring the file on disk rather than the certificate actually being served.
• Forgetting that mail servers, load balancers, and containers terminate TLS too, and need their own renewal path.
• Adding a subdomain to the web server config and assuming the existing certificate covers it.

Best practices that hold up over time

• Automate issuance and renewal from day one, even for a single domain. The habit matters more than the scale.
• Always use a deploy hook, and always reload rather than restart.
• Alert on days remaining, measured externally, with a threshold at least twice your renewal window.
• Keep an inventory of every hostname you hold a certificate for. You cannot monitor what you have forgotten about.
• Prefer DNS-01 with a provider plugin if you need wildcards, and never rely on manual DNS validation.
• Set CAA records deliberately so you know which CAs are permitted to issue for your domain.
• Test with a non-browser client before calling a certificate deployment finished.

Frequently asked questions

Why does my site work in Chrome but fail in curl?

Almost always an incomplete chain. Browsers fetch missing intermediate certificates automatically using the Authority Information Access extension; curl does not. Point your web server at the full chain file rather than the leaf certificate and both will work.

What does “unable to get local issuer certificate” actually mean?

The client could not build a path from your certificate to a root it trusts. Nine times out of ten the server is not sending its intermediates. Occasionally the client’s own CA bundle is out of date, which is common in older container base images.

Does a wildcard certificate cover the root domain?

No. *.example.com matches a single label, so it covers www.example.com but not example.com and not a.b.example.com. Include the apex explicitly as an additional SAN entry when you request the certificate.

My certificate renewed but the site still shows it as expired. Why?

The service holding the certificate has not reloaded. The new file is on disk; the old one is still in memory. Add a deploy hook that reloads the service on renewal, then verify from outside with openssl s_client rather than trusting the file timestamp.

How short will certificate lifetimes actually get?

Under the CA/Browser Forum schedule, 200 days applies from March 2026, 100 days from March 2027, and 47 days from March 2029. Private and internal CAs are not bound by these rules, but public certificates are.

Is the Common Name field still used at all?

Not for hostname validation. Modern clients read only the Subject Alternative Name extension. A certificate whose CN matches but whose SAN list does not include the hostname will be rejected.

Wrapping up

TLS certificate errors feel opaque mostly because the error text points at the wrong layer. Once you sort them into chain, SAN, or expiry, each one has a short diagnostic path and an obvious fix. Get comfortable with openssl s_client and openssl x509 and you will resolve most incidents faster than it takes to find the right dashboard.

The expiry category is the one worth engineering away permanently. With maximum lifetimes dropping toward 47 days, any process that depends on a person remembering something will fail eventually. Automate the renewal, add the deploy hook, and alert on what is actually being served rather than what is sitting on disk. Do that once and TLS certificate errors go back to being a rare curiosity instead of a recurring outage.

Need help with TLS, certificates, or infrastructure automation?

I work with teams on exactly this kind of problem — broken certificate chains, ACME automation across multiple servers, expiry monitoring wired into Prometheus and Grafana, and the wider DevOps plumbing that keeps it all running quietly in the background.

If you would rather hand this off than debug it at two in the morning, you can hire me on Upwork.

The post TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation appeared first on John Nessime.