The fix looked obvious. Drop in a limit_req_zone, apply it, reload, done. Except the reload changed nothing at all, and the reason took an embarrassingly long time to find.

This post covers Nginx rate limiting and bot mitigation the way it actually behaves in front of a real site: how the limiting modules key requests, what burst and nodelay really do to your traffic shape, why a limit that looks correct in the config can be doing nothing (or throttling everyone at once), and how to layer bot handling so you are not playing whack-a-mole with user agent strings forever.

The failure that hides: your limit is keyed on the wrong address

This is the one that bites hardest, because the config parses, nginx -t passes, and the error log stays quiet. Nothing tells you it is wrong.

If anything sits in front of Nginx (Cloudflare, another CDN, a load balancer, or a local Varnish tier), then $remote_addr is the address of that proxy, not the visitor. Key a rate limit on it and you get one of two outcomes, both bad:

• Every visitor arriving through a given edge node shares a single bucket. One aggressive client burns the budget and legitimate users start collecting 503s.
• Or your rate is generous enough that the shared bucket never fills, in which case the limit is decorative and the abusive client sails straight through.

The real IP module fixes this, but only if you trust the right sources. It rewrites $remote_addr and $binary_remote_addr from a header, and only for connections coming from addresses you have explicitly declared trustworthy.

# http context. One entry per CDN range you actually receive traffic from.
set_real_ip_from 203.0.113.0/24;
set_real_ip_from 2001:db8::/32;

real_ip_header CF-Connecting-IP;   # or X-Forwarded-For for a generic proxy
real_ip_recursive on;

Two things matter here more than the syntax.

First, never widen set_real_ip_from to 0.0.0.0/0. That tells Nginx to believe a client-supplied header from anyone, which means an attacker can pick a new “client IP” on every request and your per-IP limit becomes unlimited. It also poisons your logs, your deny rules and anything downstream that trusts the forwarded address.

Second, after the real IP module has run, $binary_remote_addr holds the visitor address and $realip_remote_addr holds the original connecting address (the proxy). Key your limits on $binary_remote_addr. Keying on $realip_remote_addr puts you right back in the shared-bucket failure, silently.

The proxy ranges change over time, so treat the trust list as a file that gets regenerated on a schedule and reloaded, not something you paste in once. On a self-managed VPS from a provider like Contabo or InterServer, a small cron job that fetches the current ranges, rewrites an include file, runs nginx -t, and only reloads on success is about twenty lines of shell and saves you from a very confusing afternoon later.

What limit_req actually does, and what burst really means

Nginx rate limiting for requests uses a leaky bucket. The zone stores, per key, the current number of excessive requests. The rate defines how fast that excess drains. Two directives do the work: limit_req_zone declares the bucket in the http context, limit_req applies it in http, server or location.

http {
    limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m   rate=20r/m;

    limit_req_status 429;
    limit_req_log_level warn;
}

The rate is expressed in requests per second (r/s) or requests per minute (r/m). If you want less than one request per second, use r/m. There is no native per-hour or per-day unit in the stock module, which surprises people writing password-reset limits.

burst without nodelay is a queue, not a rejection

This is the second-most common misunderstanding after the keying problem. By default, excessive requests are delayed so that they come out at the configured rate. They are only rejected once the number waiting exceeds burst.

location /search/ {
    limit_req zone=general burst=20;
}

At 10r/s, a client that fires 21 requests at once gets one served immediately, nineteen held and released over roughly two seconds, and the last one rejected. Those nineteen held requests are still open connections on your server. You have not reduced concurrency, you have converted a spike into a slow trickle and paid for it in worker slots and file descriptors.

That is genuinely useful for smoothing traffic to a fragile upstream. It is a bad idea for abuse control, where holding the connection is exactly what the attacker wants.

nodelay, delay, and picking between them

nodelay changes the behaviour to: serve everything in the burst window immediately, reject everything past it. The bucket still drains at the configured rate, so a client that sustains the burst rate will hit the wall and stay there.

location /wp-login.php {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    # ...fastcgi_pass or proxy_pass here
}

The delay parameter is the middle ground: it sets the point within the burst at which requests start being delayed rather than served immediately. burst=20 delay=10 means the first ten excess requests go straight through, the next ten get queued, and anything beyond is rejected.

How I decide: if a human clicking around your site could plausibly generate the spike, use nodelay or a delay that covers it. A single page load pulling twelve uncached assets through Nginx is one “request” to the user and twelve to the limiter. If the endpoint is machine-facing and expensive (search, export, login, an API write path), tighten the rate, keep the burst small, and use nodelay so rejection is fast and cheap.

Also set limit_req_status 429. The default is 503, which tells clients and monitoring that your server is broken rather than that they are being throttled. 429 is the correct semantic, and well-behaved clients back off on it.

Sizing the zone, and the number most guides get wrong

You will see “10m holds about 160,000 IP addresses” repeated everywhere. That is the 32-bit figure. Nginx documents the stored state for limit_req as 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms, so one megabyte holds roughly 16 thousand states on 32-bit and roughly 8 thousand on 64-bit. On the 64-bit server you are almost certainly running, a 10m zone is closer to 80,000 states.

This matters because of what happens when the zone fills. Nginx evicts the least recently used state. If it still cannot allocate, the request is terminated with your configured error status. A zone that is constantly thrashing under a distributed crawl will start returning errors for reasons that have nothing to do with the rate you configured. If you are limiting a high-cardinality key, give the zone room.

limit_conn solves a different problem, and you probably need both

limit_req caps how fast requests arrive. It does nothing about a client that opens sixty connections and reads each response one byte at a time. That is a concurrency problem, and it belongs to limit_conn.

http {
    limit_conn_zone $binary_remote_addr zone=perip:10m;
    limit_conn_status 429;
}

server {
    limit_conn perip 20;

    location /downloads/ {
        limit_conn perip 4;
        limit_rate_after 2m;
        limit_rate 512k;
    }
}

Three behaviours worth knowing before you pick a number.

• Not every connection counts. Nginx only counts a connection once the full request header has been read and the request is being processed. A client stalling mid-header is not counted here, which is why client_header_timeout and client_body_timeout still matter.
• Under HTTP/2 and HTTP/3, each concurrent request counts as a separate connection. A limit of 4 that felt generous on HTTP/1.1 can break a modern browser loading a page with many assets. This is a very common self-inflicted outage.
• The limit_conn state is smaller than the limit_req state, so the same zone size holds roughly twice as many entries.

The pairing is the point: limit_req protects against volume, limit_conn protects against occupancy. Rate limiting alone leaves you open to slow-read exhaustion, and connection limiting alone does nothing about a client making one fast request after another on a keepalive connection.

Choosing keys: IP is a starting point, not the answer

The key can be any variable or combination of variables. That flexibility is where most of the value is, because per-IP limits have a real accuracy problem in both directions.

On one side, carrier-grade NAT, corporate egress, university networks and shared VPN exit nodes (NordVPN, Surfshark and every other provider works this way) put many genuine users behind one address. A tight per-IP limit will hit them. On the other side, a distributed scraper renting a few hundred cheap addresses never touches a per-IP limit at all, because no single address is doing much.

So layer keys instead of tuning one harder:

limit_req_zone $binary_remote_addr zone=perip:20m    rate=10r/s;
limit_req_zone $server_name        zone=persite:1m   rate=200r/s;

Applied together, the per-IP zone catches the loud individual and the per-site zone caps total damage regardless of how many addresses are involved. When several limit_req directives match, all of them are evaluated and the most restrictive result wins. If any one of them rejects, the request is rejected.

One inheritance rule catches people out: limit_req directives are inherited from the enclosing level only if the current level defines none. Add a single limit_req inside a location and you have dropped every limit set at server level for that location. If you want both, restate both.

Allowlisting with an empty key

Requests whose key evaluates to an empty string are not accounted at all. That is the clean way to exempt monitoring probes, your office, or a payment webhook source without maintaining a parallel set of locations.

geo $limit {
    default        1;
    10.0.0.0/8     0;
    192.0.2.10/32  0;
}

map $limit $limit_key {
    0 "";                    # empty key, never limited
    1 $binary_remote_addr;
}

limit_req_zone $limit_key zone=perip:20m rate=10r/s;

Nginx evaluates map variables only when they are used, so declaring a large map costs nothing on requests that never reference it.

Roll it out in dry run, not in production

Both modules have a dry run mode. Excessive requests are counted in the shared memory zone and logged exactly as they would be, but nothing is actually rejected or delayed. Combined with the status variables, this gives you a real answer to “would this limit have hurt anyone” before you find out from a customer.

log_format ratelimit '$remote_addr $status "$request" '
                     'req=$limit_req_status conn=$limit_conn_status '
                     'ua="$http_user_agent"';

server {
    limit_req_dry_run  on;
    limit_conn_dry_run on;

    access_log /var/log/nginx/ratelimit.log ratelimit;
}

$limit_req_status reports PASSED, DELAYED, REJECTED, DELAYED_DRY_RUN or REJECTED_DRY_RUN. $limit_conn_status reports PASSED, REJECTED or REJECTED_DRY_RUN. A numbered rollout that has never embarrassed me:

1. Add the zones and the limit_req directives with dry run on, plus the log format above.
2. Run for a full traffic cycle. A week if you have weekly patterns, longer if you have a monthly billing spike.
3. Count the would-be rejections by user agent and by address. If real browsers appear, your rate or burst is wrong, not your users.
4. Turn dry run off on the least critical location first. Watch 429 rates alongside your normal error budget.
5. Only then apply it to login, checkout, or anything that costs you money when it breaks.

Shipping the rate limit log into Loki and graphing it in Grafana makes step three take minutes instead of an evening with awk. Any log platform with decent label cardinality handling works; the point is having rejection counts broken down by zone and user agent, over time, next to your latency graphs.

Bot mitigation in layers, cheapest first

Rate limiting treats all clients the same. Bot mitigation is about deciding that some clients deserve different treatment before the limiter ever sees them. The useful mental model is a set of layers ordered by cost and by how easy each is to defeat.

Layer one: robots.txt, and being honest about what it does

robots.txt is a request, not a control. Crawlers that respect it will honour a Disallow on your search endpoint, your faceted filter URLs and your infinite calendar pages, and that alone removes a surprising amount of load. Crawlers that do not respect it will ignore the file completely, and some will read it specifically to find out where the interesting URLs are.

Use it to shape well-behaved traffic. Never use it as a security boundary. And be careful about blanket disallows on search engines you actually want: deindexing yourself is a much more expensive mistake than a bit of crawler load.

Layer two: user agent classification with map

A map on $http_user_agent is cheap, readable, and easy to revise. It is also trivially spoofable, so treat it as traffic shaping rather than enforcement.

map $http_user_agent $ua_class {
    default          human;
    ""               empty;
    "~*bot|crawler|spider|scrape"  bot;
}

map $ua_class $bot_limit_key {
    bot   $binary_remote_addr;
    empty $binary_remote_addr;
    human "";
}

limit_req_zone $bot_limit_key zone=bots:10m rate=30r/m;

That gives declared bots a much tighter budget than browsers while leaving human traffic on the general limit. An empty user agent is worth its own class: almost nothing legitimate sends one, and it is a reliable signal for cheap scraping tools.

Two cautions. Broad substring matching on bot will catch strings you did not intend, so review a sample of your own logs before deploying the pattern. And blocking rather than throttling is a bigger decision than it looks: if you return 403 to a crawler you actually want, you find out weeks later in your search traffic, not in your error log.

Layer three: verifying bots that claim a name

The interesting case is a request claiming to be a major search crawler. Impersonating one is the standard way to bypass a user agent based allowlist, so a claim needs verification.

The method the search vendors themselves document is a forward-confirmed reverse DNS check: reverse-resolve the client address, confirm the resulting hostname belongs to the vendor’s domain, then forward-resolve that hostname and confirm it returns the original address. An attacker cannot fake this without controlling the vendor’s DNS. Several vendors also publish their crawler IP ranges as machine-readable lists, which avoids the DNS round trip entirely.

Here is the honest limitation: stock Nginx has no directive that does this. There is no built-in reverse DNS verification in the core modules. Your realistic options are a third-party module, doing the verification out of band and feeding the result into a geo block or an included allowlist file, or pushing the decision up to a CDN or WAF layer that already does it. Anyone showing you a pure stock-Nginx config that “verifies Googlebot” is verifying a string, not an identity.

This is also the point where an edge provider earns its keep. Cloudflare, Fastly and similar edge platforms maintain verified bot lists and fingerprinting that no single origin server can reproduce, and they drop the traffic before it costs you a worker. If your bot problem is large and distributed, the edge is the right layer and Nginx is your backstop, not your front line.

Layer four: escalation to the firewall

A client that collects hundreds of 429s and keeps going is not going to stop. Serving it a rejection still costs you a TCP handshake, a TLS negotiation and a log line. At that point the cheapest response is to stop answering at all.

Fail2ban is the usual tool: a filter that matches your rate limit rejections in the Nginx error or access log, an action that inserts a firewall drop, and a ban time long enough to be inconvenient. Two things to get right. Match on a pattern that only fires for real limit rejections, or you will ban yourself during a deploy. And if you are behind a CDN, your firewall sees the CDN address, not the client, so the ban has to be applied at the edge instead. Banning a Cloudflare range at your origin firewall takes your whole site offline, and it is a mistake people make exactly once.

Return code 444 is worth knowing here. It is Nginx-specific and closes the connection without any response, which is cheaper than generating an error page for a client that will never read it.

Troubleshooting Nginx rate limiting

The limit does nothing. Check the key first. Log $binary_remote_addr alongside $realip_remote_addr and confirm they differ in the way you expect. Then check inheritance: a limit_req in the matched location discards the server level ones. Confirm which location actually matched by adding $request_uri and a marker header while testing.

Real users are getting 429 or 503. Turn dry run back on and read the log rather than guessing. Usually it is one of three things: a burst too small for a normal page load’s asset count, a limit_conn value set for HTTP/1.1 assumptions on an HTTP/2 server, or a shared-egress network where many people genuinely share one address.

Nothing appears in the error log. The default log level for rate limit refusals is error, and delays are logged one level below whatever you set. If you set limit_req_log_level warn, delays go to info, which your log level may be filtering out entirely. Grep for the limiting messages directly:

grep 'limiting requests' /var/log/nginx/error.log | tail -50
grep 'limiting connections' /var/log/nginx/error.log | tail -50

Intermittent errors under heavy load with no matching rate. Suspect zone exhaustion. Enlarge the zone and see whether the pattern changes. High-cardinality keys and distributed crawls are the usual cause.

Health checks and uptime probes are being throttled. Add them to the allowlist map. Monitoring hitting a rate limit produces alerts about an outage that is not happening, which is worse than no monitoring.

Common mistakes

• Trusting forwarded headers from every source, which turns a per-IP limit into no limit at all.
• Leaving limit_req_status at the default 503, so throttling looks like an outage in your own monitoring.
• Using burst without nodelay on an abuse path, and quietly filling worker slots with held connections.
• Setting a limit_conn value based on HTTP/1.1 intuition on a server serving HTTP/2 or HTTP/3.
• Adding one limit_req in a location and silently dropping every server-level limit for that path.
• Blocking user agent substrings without checking a sample of real logs first.
• Applying identical limits to reads and writes. A cacheable GET and a POST that writes to the database do not deserve the same budget.
• Banning CDN address ranges at the origin firewall because the logs show them as the client.

Best practices worth the effort

• Verify the key before tuning the rate. Every other decision depends on it being correct.
• Always start in dry run and measure a full traffic cycle before enforcing.
• Use separate zones per traffic class. Search, login, API, static and general traffic have different shapes and should not share a bucket.
• Pair limit_req with limit_conn. The first caps arrival rate, the second caps occupancy, and neither substitutes for the other.
• Layer a per-server zone above your per-IP zone so distributed traffic still hits a ceiling.
• Allowlist with an empty key rather than duplicating location blocks.
• Export $limit_req_status to your log platform and graph rejections next to latency and upstream errors.
• Keep CDN trust ranges in a generated include file that is refreshed on a schedule and validated with nginx -t before reload.
• Throttle before you block. Blocking failures are invisible until they show up in your search traffic.

Frequently asked questions

Does Nginx rate limiting stop a DDoS attack?

It helps with application-layer floods that reach your server, because rejecting a request is far cheaper than processing it. It does nothing about volumetric attacks that saturate your network link, since that traffic never gets to Nginx to be limited. For that you need capacity in front of the origin: a CDN or scrubbing service. Treat rate limiting as protection against abuse and accidental floods, not as DDoS mitigation.

What rate should I actually set?

There is no universal number, and any guide that gives you one has not seen your traffic. Measure instead: run dry run mode, look at your real per-IP request distribution, and set the rate above the busiest legitimate client with headroom. Then set a separate, much tighter zone for the specific endpoints that are expensive or attacked.

Should I use 429 or 503 for rejected requests?

429 in nearly every case. It communicates that the client should slow down, well-behaved clients back off on it, and it keeps your 5xx metrics meaningful. The default is 503 purely for historical reasons.

Can Nginx rate limit per hour or per day?

Not with the stock module. It accepts requests per second and requests per minute only, so the slowest native rate is one request per minute expressed as 1r/m. Longer windows need a third-party module, a shared store such as Redis via a scripting module, or enforcement in the application itself. For something like a password reset limit, the application is usually the better place anyway, because it can key on the account rather than the address.

Do rate limit zones work across multiple Nginx servers?

Not in the open source build. Each instance keeps its own shared memory zone, so a client spread across four load-balanced servers effectively gets four times the budget. The sync parameter on limit_req_zone synchronises state across a cluster, but it is part of the commercial subscription. Without it, either divide your intended rate by your instance count, ensure your balancer pins clients consistently, or move the limit to a shared store.

Will blocking AI crawlers hurt my SEO?

Blocking training crawlers is separate from blocking search indexing crawlers, and the distinction is worth checking per vendor before you write a rule, because some vendors run several crawlers with different purposes under related names. The risk is collateral damage: a broad user agent pattern that also matches a search crawler will remove you from results, and you will not notice quickly. Verify your pattern against real log samples, and prefer throttling over outright blocking while you are unsure.

Where should rate limiting live: the CDN, Nginx, or the application?

All three, doing different jobs. The CDN handles volume and known-bad reputation before it costs you anything. Nginx handles per-client shaping and protects your workers from anything that gets through. The application handles limits that need identity or business context, like per-account quotas, which Nginx cannot see. Skipping the Nginx layer because you have a CDN leaves you exposed the moment someone finds your origin address.

The one thing worth remembering

Nginx rate limiting is not hard to configure. It is easy to configure into a state where it silently does nothing, or silently punishes the wrong people, and both failures look identical from the outside: a config that parses, a clean nginx -t, and no complaints until there are.

So get the key right, prove it with a log line rather than assuming it, and run every new limit in dry run until the data tells you it is safe. Layer bot handling by cost, throttle before you block, and be honest about which layer can actually verify an identity. Everything else is tuning, and tuning is easy once the foundation is not lying to you.

Need a second pair of eyes on your Nginx edge?

Most of the work here is diagnostic rather than clever. Things I regularly help with:

• Auditing an existing limit_req and limit_conn setup to find out whether it is actually limiting anything, including the keying and inheritance traps above.
• Fixing real client IP handling behind Cloudflare or another CDN, with a generated trust list that refreshes safely instead of drifting.
• Designing per-endpoint zones for WordPress, WooCommerce or API origins so login, search and write paths get budgets that match their real cost.
• Running a measured dry run rollout and reporting back what would have been rejected, broken down by user agent and network, before anything is enforced.
• Building bot classification and escalation: user agent maps, allowlists, Fail2ban integration, and knowing which decisions belong at the edge instead.
• Wiring rate limit telemetry into Grafana and Loki so throttling is something you can see on a dashboard rather than discover from a ticket.

If you would like a look at yours, send me the relevant server block, a sample of the access log, and whatever is currently going wrong. That is usually enough to say something useful straight away.

The post Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users appeared first on John Nessime.

So you check the zone. There is an MX record. There is an SPF record. There is a DKIM key published under a selector. There is a DMARC record. You paste the domain into three free checkers and every one of them lights up green.

Everything is configured. The domain is still trivially spoofable.

This is the failure mode that makes business email DNS worth writing about. It is not that people forget to publish records. It is that publishing a record and enforcing a policy are different acts, the checkers reward the first one, and nothing anywhere in DNS will tell you that you stopped halfway.

What follows is what each of the four records actually does, what breaks when it is wrong, the domain architecture decision that stops the problem recurring, and an order of operations that gets you to enforcement without losing legitimate mail on the way.

The two jobs your business email DNS actually does

Almost every confused email ticket traces back to one misunderstanding: people treat these four records as a single feature called “email setup.” They are two unrelated systems that happen to share a zone file.

• Receiving mail. This is MX, and only MX. Nothing else in the set participates.
• Authorizing senders. This is SPF, DKIM and DMARC. None of them affect whether a single message reaches your mailbox.

The consequence is blunt. If inbound mail is not arriving, editing SPF cannot help and never will. If your domain is being spoofed, a perfect MX configuration contributes nothing. It is common to see an afternoon spent rewriting an SPF record for a problem that lived entirely in a mail routing rule.

A third job hides inside the second, and it is the one people skip. SPF and DKIM only produce evidence: this server was authorized, this content was signed by this domain. Neither tells a receiving server what to do when the evidence is missing. Only DMARC does, through its policy tag, and that tag is the single enforcement dial in the whole arrangement.

MX: the only record inbound delivery depends on

An MX record carries exactly two values: a preference number and a hostname. Lower preference numbers are tried first. Equal preference numbers let a sender spread load across the listed hosts, which is how most hosted mail providers publish theirs.

dig +short MX example.com

The hostname side has two constraints panels will happily let you violate. The target must be a name resolving to A or AAAA records, not an IP typed into the field, and it must not point at a CNAME. Some panels accept an IP and quietly invent a hostname behind the scenes, which works until you move providers and find the record you thought you published is not the one that exists.

If a domain has no MX record at all, the SMTP specification tells senders to fall back to its A or AAAA address. On a business domain that is your web server. The connection times out or is refused, and the sender queues and retries for days before anyone sees a bounce. The mail is not rejected; it is late, sometimes by a working week.

And subdomains do not inherit MX records from the parent. If example.com has MX records, mail.example.com does not. It falls back to its own A record, or to nothing.

Null MX for domains that should never receive mail

Most businesses own more domains than they use for mail: typo-squat defenses, old brand names, a marketing microsite, the domain from the company they acquired. RFC 7505 defines a way to say “this domain accepts no mail” explicitly, using a single MX record with preference 0 and a target of a lone dot.

example-parked.com.  3600  IN  MX  0  .

A conforming sender sees this and rejects immediately instead of falling back to the A record and retrying for days. The bigger benefit is backscatter: when someone forges a return address at a domain of yours, bounce messages get aimed at it. A null MX turns those into an instant rejection at the sender’s end.

One rule attached to this: a domain advertising a null MX must not publish any other MX records. It is the whole set or nothing.

One operational note: drop MX TTLs to a few minutes before a mail migration and raise them afterwards. A low TTL costs a little query volume. A high TTL during a cutover costs you mail delivered to a server you already decommissioned, with no error anywhere to tell you. Per-record TTL control and fast propagation matter here, which is why I would rather do a cutover on something like Cloudflare than a registrar’s default panel.

SPF: an authorization list with a hard budget

SPF is a TXT record at the domain apex that begins with v=spf1 and lists the systems allowed to send with your domain in the envelope sender. Publish two of them and you have published none: the result is a permanent error, which receivers treat as a failure.

dig +short TXT example.com

The constraint everyone eventually hits is the lookup budget. Evaluation is capped at ten DNS-querying mechanisms. The terms that count are include, a, mx, ptr, exists and redirect, and the count follows nested includes all the way down, so one vendor include: can carry four lookups inside it. Literal ip4 and ip6 ranges cost nothing, and neither does all.

Exceed ten and evaluation stops with a permanent error. It does not fail open. It does not keep counting and pass anyway.

There is a second, quieter limit: void lookups, meaning queries that return nothing. Implementations are told to cap those at two. This is the one that bites long after you stop paying attention, because the trigger is a vendor you cancelled whose include: target eventually stopped resolving. Your record did not change. Your mail started failing anyway.

Over budget, the honest options are to drop vendors you no longer use, replace lookup-heavy mechanisms with the literal IP ranges they resolve to, or move senders onto their own subdomains so each gets a fresh budget. Flattening services that resolve includes and keep the record updated do work, but understand the trade: you have taken on a dependency that must track your vendors’ IP changes, and cloud senders rotate ranges.

Why hard fail is not automatically the safer choice

SPF records end with a qualifier on all. Soft fail is ~all; hard fail is -all. The instinct is that hard fail is the stricter, more secure choice, and it is, but the DMARC specification calls out a specific cost that is easy to miss.

SPF is evaluated early in the SMTP conversation. A hard fail can get a message rejected before the message body is ever transmitted, which means before the From header is visible and before DMARC is evaluated at all. A message that would have passed DMARC on an aligned DKIM signature can be rejected purely on SPF.

Messages rejected that early never reach the DATA phase, so they never appear in your DMARC aggregate reports. You lose the failure and the evidence of the failure in the same move.

That is the argument for staying on ~all while you are still discovering who sends as you, and moving to -all once your reports have gone quiet. On a parked domain with no legitimate senders at all, publish v=spf1 -all immediately; there is nothing to discover.

One more thing SPF does not do, and this is the root of most DMARC confusion. SPF authenticates the envelope sender, the address in the SMTP MAIL FROM command. Your recipients never see that address. They see the From header, which SPF does not check at all.

DKIM: the signature that survives a forward

DKIM adds a cryptographic signature over selected headers and the message body. The public key lives in DNS under a selector, in the form selector._domainkey.example.com, and the signature header names both the selector and a signing domain in its d= tag.

dig +short TXT s1._domainkey.example.com

That d= value is the part that matters and the part that gets ignored. A message can carry a perfectly valid signature from your provider’s own domain. It verifies. It proves nothing about you, because the identity it asserts is theirs. Completing your provider’s domain authentication step, the one that has you publish records under your own zone, is what moves d= onto your domain.

Algorithms and key size

The rules here are settled. Signers must use rsa-sha256. The older rsa-sha1 must not be used for signing or verifying, and signatures using it fail permanently. Keys must be at least 1024 bits and should be at least 2048. Verifiers are required to reject anything under 1024 outright.

A useful field heuristic follows from a DNS quirk. A single TXT character string maxes out around 255 characters, and a 2048-bit RSA key does not fit in one. So if your DKIM record comes back from dig as one quoted string, it is probably 1024-bit; split into two concatenated strings, probably 2048. A hint rather than proof, but it takes two seconds.

An Ed25519 signing algorithm is also defined for DKIM, with compact keys and good cryptographic properties, but support across platforms and verifiers is still uneven. RSA with SHA-256 remains the safe universal choice unless you sign with both.

Rotating a key without breaking anything

Never edit a DKIM key in place. Mail already in flight, in a queue, or sitting in a mailing list’s outbound spool was signed with the old key, and overwriting the record invalidates all of it retroactively.

1. Publish the new key under a new selector name and confirm it resolves.
2. Switch the signing platform to the new selector.
3. Wait out the longest plausible delivery delay, which is days rather than hours if mailing lists are involved.
4. Remove the old selector record.

DKIM has one property SPF does not, and it decides your architecture later. A signature travels with the message. When mail passes through a forwarder, a role alias or a mailing list, the connecting IP is no longer yours and SPF fails; the DKIM signature usually still verifies. This is why the DMARC specification states that domains publishing a reject policy must not rely on SPF alone and must apply valid DKIM signatures.

DMARC: connecting the visible domain to the proof

DMARC is a TXT record at _dmarc.example.com. It asks one question that neither SPF nor DKIM asks: does the domain your recipient can actually see match the domain that proved itself?

dig +short TXT _dmarc.example.com

That match is called alignment. Relaxed alignment, the default, accepts any domain sharing the same organizational domain, so mail.example.com aligns with example.com. Strict requires them to be identical. A message passes DMARC if either SPF or DKIM passes and the domain that passed aligns with the From header. One aligned mechanism is enough. Both passing while neither aligns is a fail, and that is the most common cause of “but all my checkers are green.”

The tags worth knowing

• p is the policy for the domain itself: none, quarantine or reject. Omit it and the record is treated as p=none.
• sp sets a separate policy for existing subdomains.
• np sets a policy for subdomains that do not exist in DNS at all. This is the anti-spoofing tag most zones are missing.
• rua is where aggregate reports are sent. Without it you get no reports, and without reports you are guessing.
• adkim and aspf switch the two alignment modes from relaxed to strict.
• t=y is test mode. It tells receivers to apply one level below your stated policy, so a record saying reject behaves as quarantine, and quarantine behaves as none.

The DMARC specification has been rewritten and promoted onto the IETF standards track as RFC 9989, with aggregate and failure reporting split into RFC 9990 and RFC 9991, obsoleting the original RFC 7489. Existing records keep working and still begin with v=DMARC1, so nothing is urgent. Three things are worth acting on at your next zone edit.

The pct, rf and ri tags are gone. Unknown tags are ignored rather than treated as errors, so leaving them does no damage, but pct no longer means anything and the staged rollout it was used for is now t=y.

The Public Suffix List is no longer how receivers determine your organizational domain. They now walk up the DNS hierarchy looking for a DMARC record at each level, capped at eight queries. For a two-label domain this changes nothing. For a deep subdomain hierarchy it can resolve differently than the old method did, and a name deeper than eight labels will never have its policy record found by the walk. If you run a complex namespace, publish an explicit record at the exact name rather than relying on inheritance.

And if two DMARC TXT records are returned for the same name, both are discarded. A duplicate is not a merge or a tiebreak. It is the same as having none.

Reading the reports

Aggregate reports arrive as gzipped XML, typically once a day per reporting provider. They are technically human-readable and practically not, especially once a dozen providers are reporting on a domain with several sending services.

Two honest choices. Parse them yourself, which is a small satisfying project that gives you exactly the view you want. Or point rua at a hosted processor; Dmarcian, Valimail and PowerDMARC all do this competently, with entry tiers sufficient for one domain. The trade is that you are sending metadata about your mail flows to a third party and inheriting their retention terms. Read those terms first in a regulated environment.

Domain architecture: the decision that stops this recurring

Everything above is records. This part is structure, and it is what separates a zone that stays correct from one that quietly rots.

Put each sending stream on its own subdomain. People mail from the apex. Marketing sends from something like news.example.com. Transactional and application mail sends from mail.example.com or similar. Each subdomain gets its own SPF record, its own DKIM selectors, and its own DMARC record if you want a different policy there.

• Each subdomain gets a fresh ten-lookup SPF budget, which is usually what forced the decision in the first place.
• A marketing platform that gets breached or blocklisted does not drag your invoices and password resets down with it.
• Reputation is scoped. Bulk sending and one-to-one business mail get judged separately, which is what you want, because they behave nothing alike.
• You can hold the apex at a strict policy while a subdomain is still being brought into line.

The cost is more records to maintain and one more thing to explain to whoever inherits the zone. It is worth it the first time a vendor migration happens and you only have to touch one subdomain.

Then close the gaps. Every domain that never sends and never receives should carry a null MX, an SPF record of v=spf1 -all, and DMARC at p=reject. Three records, five minutes, and the domain leaves the pool of things abusable in your name. Add np=reject on live domains too, because an attacker does not need a subdomain to exist in order to put it in a From header.

Rolling it out without losing mail

The order matters, because each step depends on evidence produced by the one before it.

1. Inventory every sender. Mailbox provider, CRM, helpdesk, billing system, marketing platform, monitoring alerts, and the tool a department signed up for without telling anyone. This list is always longer than expected and it is the highest-value step here.
2. Fix MX first, with a low TTL if you are migrating. Inbound is what people notice within minutes.
3. Publish SPF within budget, ending in ~all for now. Count lookups before publishing, not after mail fails.
4. Get DKIM signing on your own domain for every stream. Verify d= in the headers of a real received message rather than trusting a dashboard.
5. Publish DMARC at p=none with a rua address. This changes nothing about delivery and starts the flow of evidence.
6. Read reports for several weeks. Longer if you have quarterly patterns like invoicing runs or annual renewals, which is exactly the mail you cannot afford to break.
7. Move to p=quarantine once every remaining failure is one you recognize as spoofing. Use t=y for a softer first step.
8. Move to p=reject, confirming DKIM is aligned on every stream first, then tighten SPF to -all.

Step six is the one people compress, and it is the only step that generates the information the last two depend on. Skipping it means going to enforcement blind and finding out which stream you broke from a colleague rather than from a report.

Steps one and six are also the two that never finish. Keep the sender inventory current as vendors come and go, and keep reading reports after you reach enforcement, because a new tool bought by a department you never talk to shows up in a report long before it shows up in a conversation.

Troubleshooting, in the order worth checking

When something is wrong, the first question is which of the two jobs failed. Inbound problems and spoofing problems share no diagnostic steps.

Inbound mail is not arriving

1. Confirm the MX records exist and resolve, and that each target has an A or AAAA record.
2. Query your authoritative nameservers directly rather than a resolver, so you are looking at what you published rather than what is cached.
3. Confirm the mail host actually answers on port 25 and offers STARTTLS.
4. If all three pass, the problem is inside the mail platform: routing rules, a filter, an aliasing mistake, a mailbox over quota. It is no longer DNS.

dig +short NS example.com
dig @ns1.example-dns.net example.com MX
openssl s_client -starttls smtp -connect mail.example.com:25

Legitimate mail is being rejected or quarantined

1. Read the SMTP rejection text. Receivers name the reason, and DMARC-related rejections usually say so in plain words.
2. Open the headers of a message that failed and read the authentication results. They tell you which check failed and which domain was evaluated.
3. Compare the d= value in the DKIM signature against the From domain. If they differ and share no organizational domain, alignment is your problem.
4. Count your SPF lookups. A permanent error there presents as a plain SPF failure with nothing in the record that looks wrong.
5. Check the sending IP has a valid reverse DNS record that resolves forward again. Only whoever controls the IP can set this, which means your provider if you are on shared infrastructure, or you if you run your own server.

dig -x 203.0.113.10 +short

That reverse DNS point matters more than it looks. Major mailbox providers expect sending IPs to have valid forward and reverse DNS, so if you run your own mail server, PTR control is a hard requirement rather than a nice-to-have. Confirm it before you buy. Hosts aimed at self-hosting, InterServer and Contabo among them, expose PTR in the panel; plenty of budget instances do not, and some block port 25 outright.

Common mistakes

• Treating p=none as done. It is a monitoring mode. It provides no protection whatsoever, and every checker will still show you a green tick.
• Publishing two SPF records. Usually one from the mailbox provider’s setup wizard and one from a marketing tool. The result is worse than either alone.
• Assuming a passing DKIM signature is your DKIM signature. Check d=, not the presence of a signature.
• Editing a DKIM key in place instead of rotating through a second selector.
• Leaving parked and brand domains bare. No MX, no SPF, no DMARC is an open invitation, and these domains are usually the ones nobody is monitoring.
• Going to p=reject without DKIM aligned on every stream. Forwarded mail will break, and forwarded mail is disproportionately mail from customers.
• Setting rua and never reading it. Reports you do not process are the same as reports you never requested.

Frequently asked questions

Do I need both SPF and DKIM, or is one enough?

DMARC only needs one of them to pass and align. Publish both anyway. They fail in different circumstances, and having only one means any situation that breaks it breaks all your authentication at once. Major mailbox providers also expect both from higher-volume senders, and DKIM specifically is required if you intend to publish a reject policy.

How long should I stay at p=none?

Long enough to see a full cycle of your sending patterns, plus time to fix what the reports reveal. For a small business with a mailbox provider and two SaaS tools, a few weeks. For an organization with quarterly billing runs and departments who buy their own software, months. The signal to move is that you recognize every remaining failure as illegitimate, not that a calendar reminder fired.

Will DMARC stop phishing?

It stops exact-domain spoofing, one technique among several. It does nothing about lookalike domains, nothing about display-name spoofing where the visible name says your CEO but the address is free webmail, and nothing about a compromised mailbox sending properly authenticated mail. One control worth having, not a phishing solution.

Does any of this affect whether I receive email?

No. SPF, DKIM and DMARC describe mail leaving your domain. Only MX affects mail arriving at it. The one indirect connection is that your own inbound filtering may evaluate other people’s SPF and DKIM when deciding what to do with their mail, but that is your mail platform’s behavior, not your DNS records.

Do I need to update my records for the new DMARC RFCs?

Nothing breaks if you do nothing. At your next edit, drop pct, rf and ri if they are present, consider adding np to close the non-existent subdomain gap, and if you run a deep subdomain hierarchy, publish explicit records rather than relying on inheritance. Leave the rest at defaults, because the defaults are sensible.

My mail goes to spam even though everything passes. What now?

Authentication and reputation are different problems. Passing DMARC tells a receiver the mail genuinely came from you; it says nothing about whether recipients want it. If authentication is clean, look at complaint rates, list hygiene, volume ramps and content. Google publishes a spam complaint ceiling and its guidance is to operate well below that ceiling, not near it.

Wrapping up

If you take one thing from this, take the distinction between publishing and enforcing. Business email DNS gives you four records that answer four different questions, and only one of them, the DMARC policy tag, actually instructs a receiving server to do anything. Every free checker will confirm the records exist. None of them will tell you the dial is at zero.

Get MX right so mail arrives. Keep SPF inside its lookup budget. Make DKIM sign with your own domain so d= aligns. Publish DMARC at p=none with a reporting address, read what comes back for as long as it takes, then move the dial until it reaches reject.

And while you are in the zone, spend five minutes on the domains nobody thinks about. A null MX, an empty SPF record and a reject policy on a parked domain is the cheapest security work available to you.

Need a second pair of eyes on your mail domain?

Mail DNS is unusually well suited to outside help, because the failure is almost always somewhere the domain owner cannot see it: in a header, in a report nobody parses, or in a vendor’s include chain. Things I regularly help with:

• Auditing an existing zone across MX, SPF, DKIM and DMARC and telling you plainly what is exposed and what is fine
• Getting an SPF record back under the ten-lookup limit without breaking a sender you forgot about
• Planning and executing the move from p=none to p=quarantine to p=reject, including the sender inventory and the report reading
• Setting up DMARC aggregate report collection and parsing, either self-hosted or on a processor
• Splitting sending streams onto subdomains and migrating each one without a delivery gap
• Locking down parked and brand-protection domains so they stop being usable in your name

If you would rather not describe the problem in the abstract, send me the output of a dig against your domain, or a set of headers from a message that failed, and I will tell you what I see in it.

The post Business Email DNS Done Right: MX, SPF, DKIM and DMARC appeared first on John Nessime.

So you close the ticket. Then it happens again next week, to a different customer, on a different mailbox provider.

This is the shape of almost every WordPress email deliverability problem I’ve had to unpick. The site is not broken and the mail is genuinely leaving. It is being evaluated somewhere downstream and quietly filed into spam by a receiver that has decided it cannot prove who sent it. Nothing in WordPress will ever tell you this happened.

This post covers the four places that failure actually lives: how the message leaves your server, whether SPF survives evaluation, whether DKIM signs and aligns, and what DMARC does with the result. It is organised by failure family rather than by protocol tour, because the protocols are individually well documented and the failures are not.

The failure mode that costs you money: silent success

Start with the thing that makes this hard to debug. wp_mail() returns a boolean. That boolean tells you whether PHPMailer handed the message off without throwing. It tells you nothing about whether the message was delivered, and nothing at all about whether it reached an inbox rather than a spam folder.

So “WordPress says it sent” and “the customer got it” are separated by four or five systems you have no visibility into. That gap is where the money leaks: WooCommerce order confirmations, membership welcome mails, password resets, form notifications. All of them fail the same silent way.

So stop guessing about the boolean. WordPress fires an action when the handoff itself fails, and almost nobody hooks it:

// Log the real reason PHPMailer refused, instead of a bare false.
add_action( 'wp_mail_failed', function ( $error ) {
    error_log( 'wp_mail failed: ' . $error->get_error_message() );
} );

Then force a send from the command line so you are testing the mail path and not a contact form plugin:

wp eval 'var_dump( wp_mail( "you@example.com", "deliverability test", "body" ) );'

If that prints bool(true) and the mail still never arrives, you have ruled out WordPress entirely. Everything from here is transport and DNS.

Failure family one: how the message leaves the box

By default wp_mail() falls through to PHP’s mail(), which shells out to the local mail transfer agent. On a shared host or a default VPS build, that means your marketing site is now an email server with none of the reputation, warm-up, or bounce handling that implies.

Receivers are not being unreasonable about this. A message arriving from a web server IP, with no authentication, from a domain the IP has no relationship with, looks exactly like a compromised WordPress install pumping out spam. Because most of the time, it is.

The three transport options, honestly

Local mail(). Genuine case for it: zero dependencies, no credentials to leak, and it works fine when the box already runs a properly configured MTA with correct reverse DNS and a warmed IP. Where it fails: almost everywhere else. Most hosts either block outbound port 25 or run an MTA nobody has configured for authentication. Skip it unless you own the mail infrastructure and know its reputation.

Authenticated SMTP submission. WordPress connects to a real mail provider over port 587 with STARTTLS, or 465 with implicit TLS, and authenticates. This is the portable option, it works with any provider, and it is what I reach for first on a site I did not build. The cost is that SMTP is a chatty, connection-oriented protocol running inside a page request, so a slow or throttled provider makes checkout feel slow.

Provider HTTP API. The plugin posts the message to an API endpoint over 443 instead of speaking SMTP. Faster, immune to hosts that block SMTP ports outbound, and usually gives you richer delivery events back. The cost is real lock-in: your sending config is now provider-specific and moving means reconfiguring, not just changing a hostname.

On the plugin side, WP Mail SMTP and Post SMTP both cover the common providers and both have usable free tiers. For providers, transactional-focused services like Postmark, Amazon SES, Mailgun, Brevo and SendGrid all do this job well; the differences that matter are bounce webhook quality, log retention, and how quickly a human answers when your account gets throttled. If your host bundles a transactional relay, check whether it authenticates as your domain or the host’s shared domain before you rely on it. Providers like InterServer, Cloudways and Kinsta differ meaningfully here.

Prove the transport before you touch DNS

Before blaming records, confirm the server can actually reach the relay. This opens a TLS session against the submission port and shows you the SMTP banner and advertised capabilities:

openssl s_client -starttls smtp -crlf -connect smtp.example.net:587

-starttls smtp issues the STARTTLS upgrade for you, and -crlf sends proper line endings so the session does not hang. If this times out, your host is filtering outbound SMTP and you want the HTTP API transport instead of a longer argument with support.

To send a full test message end to end, swaks is the tool worth having on every box:

swaks --to you@example.com --from wp@example.com 
      --server smtp.example.net:587 --auth --tls

It prints the entire conversation, so an authentication failure or a policy rejection shows up as an actual SMTP response code rather than a shrug from a plugin.

Failure family two: SPF passes until it silently doesn’t

SPF is a DNS TXT record listing who may send using your domain in the envelope sender. It looks like this:

v=spf1 include:_spf.google.com include:sendgrid.net ~all

Two things about that record cause more WordPress email deliverability incidents than anything else in SPF.

The ten lookup budget

RFC 7208 caps SPF evaluation at ten mechanisms that require DNS lookups. include, a, mx, ptr, exists and redirect all count, and crucially so does everything nested inside someone else’s include. ip4 and ip6 cost nothing.

Cross the limit and the receiver returns permerror. That is not a soft fail. The receiver could not evaluate your record at all, treats SPF as failed, and every message from the domain is affected at once.

Here is why it bites so hard on WordPress sites specifically. Nobody adds ten includes in one sitting. Someone adds Google Workspace. Six months later the marketing team adds a newsletter platform. Then a CRM. Then a help desk. Then you add a transactional provider for the WooCommerce mail, and the record tips over. Nothing in DNS warns you, no plugin surfaces it, and the only visible signal is in DMARC aggregate reports you probably aren’t reading yet.

Check the record and count what it actually resolves to:

dig +short TXT example.com

Counting nested includes by hand is tedious and error-prone, so use a checker that expands them. When you are over budget, the durable fixes are removing senders you no longer use and moving distinct sending systems onto their own subdomains, each with their own SPF record. Flattening includes into raw IPs works, but you have just taken ownership of someone else’s IP ranges and they will change without telling you.

One record, and mind the envelope

Publish exactly one SPF record per domain. Two records is not additive, it is invalid, and receivers may fail the check outright.

Also remember SPF checks the envelope sender, not the address your visitor sees. On WordPress those can easily differ, especially when a plugin sets a friendly From address while the MTA uses the system user as the envelope. If you need to control it explicitly:

add_action( 'phpmailer_init', function ( $phpmailer ) {
    $phpmailer->Sender = 'bounces@example.com';
} );

Setting Sender sets the envelope from, which is what SPF evaluates and where bounces are returned. It is not the same field as From, and conflating the two is a classic source of “but SPF passes when I test it” confusion.

Failure family three: DKIM signs, but alignment is what counts

This is the invisible one, and if you take one thing from this post, take this.

DKIM adds a cryptographic signature to the message. The receiver fetches your public key from DNS at the selector your provider gave you and verifies the signature held. Check yours like this:

dig +short TXT selector1._domainkey.example.com

Straightforward enough. Here is the part that catches people: SPF and DKIM can both pass and DMARC can still fail.

DMARC does not care that a domain authenticated. It cares that the domain the recipient sees in the From header matches the domain that authenticated. That match is called alignment, and it is the whole point of DMARC. SPF aligns when the envelope sender domain matches the From domain. DKIM aligns when the signing domain in the d= tag matches the From domain. You need at least one of them aligned. Not both, but at least one.

Now picture the default setup on a WordPress site with a transactional provider. The From header says orders@example.com. The envelope sender is a bounce address on the provider’s own domain, so SPF authenticates the provider and does not align with you. And if you skipped the DNS records the provider asked you to add, the message is signed with the provider’s shared key, so d= is the provider’s domain and DKIM does not align either.

Both checks pass. Alignment fails. DMARC fails. Every deliverability tester you paste the record into says your DNS is fine.

The fix is nearly always the same: complete your provider’s domain authentication step so mail is signed as your own domain rather than their shared one. That usually means publishing a CNAME or TXT record at a selector under _domainkey, and often a CNAME for a custom return path so SPF aligns too.

Read the headers, don’t trust the dashboard

Send yourself a real message from the site and open the raw source. The Authentication-Results header the receiver added is the ground truth. You are looking for the SPF result, the DKIM result, the d= domain on the DKIM signature, and the DMARC verdict. If d= is not your domain and the SPF domain is not your domain, you have found your problem regardless of what any dashboard claims.

One practical note on the key itself: a 2048-bit DKIM public key does not fit in a single 255 character DNS string and has to be published as multiple concatenated strings. Most DNS providers handle this transparently. Some do not, and the symptom is a key that looks present but fails verification.

Failure family four: DMARC policy, and what just changed

DMARC ties the previous two together. It tells receivers what to do when neither SPF nor DKIM aligns, and it asks them to send you reports about it.

_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com"

Start at p=none. It changes nothing about how mail is handled and turns on the aggregate reports, which are the only view you will ever get of who is sending as your domain. Read them for a few weeks, find the legitimate senders you forgot about, fix their alignment, then move to p=quarantine and eventually p=reject.

Do not skip straight to p=reject on a domain you have not observed. You will reject your own invoicing system.

The specification itself was updated. RFC 7489, the original DMARC document from 2015, has been obsoleted by RFC 9989 together with RFC 9990 for aggregate reporting and RFC 9991 for failure reporting, and DMARC has moved onto the IETF standards track. Records still begin with v=DMARC1 and nothing you have published stops working, so this is not an emergency. Two changes are worth knowing:

• The pct tag is gone, along with rf and ri. Percentage-based staged rollout was implemented inconsistently by receivers and has been replaced by a simple t testing flag. If your record still carries a partial pct value from a rollout that never finished, that is worth cleaning up.
• Organisational domain discovery no longer depends on the Public Suffix List; receivers now walk the DNS tree. Single-domain owners will not notice. If you run a lot of subdomains, publishing explicit records rather than relying on inheritance removes the ambiguity.

There are also new optional tags. The one worth considering is np, which sets a policy for non-existent subdomains so attackers cannot spoof from a subdomain you never created, without changing your main policy.

Aggregate reports arrive as compressed XML and are unpleasant to read by hand. Feeding them to a processor is worth it; dmarcian, Postmark’s free DMARC digest and MXToolbox all do this, and Cloudflare can host the DNS side.

Are the bulk sender rules your problem?

Probably not, and it is worth saying so plainly because a lot of writing on this topic implies otherwise. Google’s stricter bulk sender requirements apply to domains sending roughly 5,000 or more messages a day to personal Gmail accounts. A typical WordPress business site sends a few dozen.

What changed for everyone is the direction of travel. Authentication that used to be a nice-to-have is now the baseline for being evaluated on your merits rather than treated as suspicious by default. If you do cross into bulk territory, the additional obligations are one-click unsubscribe on marketing mail via the List-Unsubscribe and List-Unsubscribe-Post headers, and keeping spam complaints low. Google’s published guidance is to stay under 0.1% and never reach 0.3%.

Domain strategy: keep resets away from newsletters

One structural decision saves more grief than any single record. Separate your transactional mail from your marketing mail at the domain level.

Send password resets and order confirmations from something like mail.example.com, and send campaigns from news.example.com. Each subdomain gets its own SPF record, which keeps you comfortably inside the lookup budget. More importantly, reputation is largely tracked per sending domain. When a campaign goes badly and complaints spike, the damage stays contained. Your password resets keep landing.

The trade-off is real: more DNS records to maintain, more DMARC reports to read, and a From address that is slightly less clean. On a site where a lost order confirmation costs actual revenue, that is a trade worth making.

Troubleshooting: a working order

1. Does the message leave? Hook wp_mail_failed and send via WP-CLI. A logged error here means credentials, ports or a plugin, not DNS.
2. Can the box reach the relay? openssl s_client -starttls smtp against the submission port. A timeout means outbound filtering.
3. What does the receiver say? Send to a mailbox you control, open the raw source, read Authentication-Results. This single header settles most arguments.
4. Is the DKIM d= your domain? If not, you have not completed provider domain authentication. Fix that before anything else.
5. Does SPF actually evaluate? Expand nested includes and count. A permerror looks like a pass in most naive testers.
6. What do the aggregate reports show? If mail is failing for a subset of recipients only, the reports will name the sending source you forgot about.

If Gmail is deferring rather than rejecting, the enhanced status code in the bounce tells you which check failed. Codes in the 4.7.x range point at authentication or alignment specifically rather than content, which narrows the search considerably.

Common mistakes

• Publishing two SPF records and assuming they merge. They do not.
• Treating a green tick from an online record checker as proof of delivery. Those tools validate syntax, not alignment on your actual mail.
• Jumping to p=reject before reading a single aggregate report.
• Setting a From address on a domain you do not control, such as a customer’s own address on a contact form. Use your domain in From and put theirs in Reply-To.
• Adding a new SaaS include without re-counting the lookup budget.
• Leaving a stale include for a service you stopped using two years ago, which is both a wasted lookup and a live authorisation.
• Configuring an SMTP plugin and never testing what happens when the provider is down. Mail queued nowhere is mail lost.

Best practices

• Route WordPress mail through a real provider, and prefer the HTTP API transport when the host filters SMTP ports.
• Complete provider domain authentication properly so DKIM signs as your domain. This is the single highest-value action on the list.
• Publish DMARC at p=none with rua on day one, even before you intend to enforce. The reports are the visibility you are missing.
• Keep SPF under the ten lookup budget with headroom for the next service someone adds.
• Split transactional and marketing sending onto separate subdomains.
• Enable email logging in your SMTP plugin so you have a record of what was attempted and when.
• Monitor DKIM key expiry and provider selector changes the same way you monitor TLS certificates. Both fail silently and both fail completely.

Frequently asked questions

Do I need an SMTP plugin if my host already sends mail?

Check what the host’s relay authenticates as. If it signs with the host’s shared domain rather than yours, DKIM will not align with your From address and DMARC will fail even though mail is technically going out. If it authenticates as your domain, you are fine without a plugin.

Is SPF enough on its own?

No. SPF breaks on forwarding, because the forwarding server is not in your record, and it authenticates the envelope sender rather than the visible From address. DKIM survives forwarding and gives you a second, independent path to alignment. Publish both.

Why do my emails reach Gmail but not Outlook, or the reverse?

Receivers weight signals differently and each maintains its own reputation view of your sending IP and domain. Divergent behaviour usually means your authentication is borderline rather than solid, so one receiver’s threshold catches it and another’s does not. Fix alignment rather than chasing individual receivers.

How long until DNS changes take effect?

As long as the TTL on the record, plus whatever the resolver in front of you decides to cache. Lower the TTL before you plan changes, not after. Verify with dig against an authoritative nameserver rather than trusting a browser-based checker.

Does DMARC stop people spoofing my domain?

At p=reject it stops exact-domain spoofing at receivers that honour the policy, which is most large ones. It does nothing about lookalike domains, display-name spoofing, or a compromised mailbox sending genuinely authenticated mail. It is one control, not a complete answer to phishing.

Should I update my DMARC record for the new RFCs?

Nothing breaks if you do not. At your next DNS edit, remove pct, rf and ri if present, and consider adding np if you want to close the non-existent subdomain gap. Leave the new tags unset otherwise, because the defaults are sensible.

Why did deliverability get worse after I moved hosts?

Usually because the old host had a relay configured that the new one does not, so PHP mail() is now sending direct from an unwarmed IP with no authentication. It is also common for a migration to move the site but not the DNS records the old provider had added. Re-verify SPF, DKIM selectors and the return path after any move.

Wrapping up

If you remember one thing about WordPress email deliverability, remember that authentication and alignment are different questions. SPF passing and DKIM verifying does not mean DMARC passes. The receiver is asking whether the domain your recipient can see is the domain that proved itself, and on a default WordPress plus transactional provider setup, the honest answer is often no.

Get the transport off PHP mail(), complete your provider’s domain authentication so d= is your own domain, keep SPF inside its lookup budget, and publish DMARC at p=none so you can finally see what is happening. Then read the reports for a month before you enforce anything.

The alternative is closing the same ticket every week and never learning why.

Need someone to fix this properly?

Mail problems are unusually well suited to outside help, because the failure is almost always in a place the site owner cannot see. This is work I take on regularly:

• Auditing an existing setup end to end and telling you exactly which of the four failure families you are in
• Rebuilding SPF records that have drifted past the ten lookup limit, without breaking a live sender
• Completing DKIM and return-path domain authentication with your provider so alignment actually holds
• Taking a domain from p=none to enforcement safely, using the aggregate reports rather than guesswork
• Splitting transactional and marketing sending onto separate subdomains with their own records and reputation
• Migrating WordPress mail off PHP mail() onto a provider, with logging and failure alerting wired up

If you want a second opinion, send me a raw message source with the full headers, or your current SPF and DMARC records, and I will tell you what I see.

The post Why Your WordPress Emails Vanish: SMTP, SPF, DKIM and DMARC appeared first on John Nessime.

The line you wrote was real. The file was saved. What you did not know is that your distro puts an Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, that sshd uses the first value it finds for any given keyword, and that your cloud provider’s image dropped a 50-cloud-init.conf in that directory setting PasswordAuthentication yes. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.

You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.

That is the thing most guides get wrong, and it is the idea this VPS hardening checklist is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.

This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.

Before you touch anything

Two minutes of preparation that saves an evening.

Find your provider’s console before you need it. Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.

Never close the session you are working in. Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.

Then update the system, because everything else assumes current packages:

sudo apt update && sudo apt upgrade -y
sudo reboot   # if a kernel update landed

1. A real user, and keys instead of passwords

Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.

# On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"

# On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10

Now open a second terminal and confirm ssh deploy@203.0.113.10 works, and that sudo -v works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.

2. SSH config, and why your edit might do nothing

This is the section that actually matters. Two mechanisms silently override what you write.

Drop-in files and first-match-wins

Modern OpenSSH packages ship an Include line near the top of /etc/ssh/sshd_config, which pulls in every .conf file from /etc/ssh/sshd_config.d/ in alphabetical order, before the rest of the main file is read. Combine that with sshd’s rule that the first value found for a keyword wins, and the consequence is specific: your drop-in has to sort earlier than the vendor’s, not later.

Most guides tell you to name it 99-something.conf, which is the convention almost everywhere else in Linux and exactly backwards here.

# /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy

A note on KbdInteractiveAuthentication: older guides and older OpenSSH call this ChallengeResponseAuthentication. Setting only PasswordAuthentication no can still leave a keyboard-interactive path open on some configurations, which is why both are here.

Now the part nobody does. Validate the syntax, then read the effective config:

# Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'

If passwordauthentication yes comes back, your file lost to something else in that directory. Look at what is in there with ls /etc/ssh/sshd_config.d/ and either rename yours to sort earlier or edit the offending file directly.

Socket activation, and the port that will not change

The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: ssh.socket holds the listening port and starts sshd only when a connection arrives. On those systems the Port and ListenAddress directives in sshd_config may be ignored entirely, because systemd owns the socket.

People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:

# -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp

If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under /lib/systemd/system/, which a package update will overwrite:

sudo systemctl edit ssh.socket

# In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it

Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.

3. Firewall, in the right order

The ordering here is the whole trick. Set the default policy and add your rules before you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose

Read that status verbose output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.

One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with -p 5432:5432 is reachable from the internet even though ufw says that port is denied. Bind container ports to 127.0.0.1 explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.

A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.

4. Automatic security updates

Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug

By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.

5. fail2ban, and what it is actually for

Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.

What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials do exist: a mail server, a web application login form, an FTP daemon.

sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd

Check that status output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, journalctl -u fail2ban will tell you why.

Set ignoreip in jail.local to include your office or home address. Banning yourself is a rite of passage but it is avoidable.

6. The boring items that matter more than the exciting ones

Nobody writes blog posts about these, and they cause more real outages than any of the above.

Provider account security

Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.

Time and timezone

sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"

UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.

Swap on small instances

# A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

Backups that leave the machine

Your provider’s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.

Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man’s switch service at the backup job so you find out when it stops running, rather than the day you need it.

Knowing the box is alive

At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.

What I would skip

Not everything in the long hardening guides earns its cost on a single web server.

• Large sysctl hardening blocks pasted from a gist. Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.
• AIDE, auditd and file-integrity monitoring on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.
• Rebuilding OpenSSH’s cipher list. Current defaults are good. Guides that hand you a Ciphers line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.
• Disabling ping. It breaks your own monitoring and hides nothing.

Troubleshooting

Locked out after an SSH change

Provider console, log in as root, and revert. If you cannot remember what you changed, ls -lt /etc/ssh/sshd_config.d/ shows the most recently modified file. Run sshd -t to see whether the config is even valid, since a syntax error stops sshd starting at all.

Key auth rejected, password prompt appears instead

Almost always permissions. The home directory must not be group-writable, ~/.ssh needs 700 and authorized_keys 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run ssh -v to see how far it gets, and check journalctl -u ssh on the server for the actual reason.

The setting I applied is not in effect

sudo sshd -T and search for the keyword. If the effective value differs from your file, something in sshd_config.d/ sorts earlier, or a Match block later in the config is overriding it for your connection.

A service is reachable that the firewall says is blocked

Docker, or another daemon writing netfilter rules directly. Check with sudo iptables -S or sudo nft list ruleset, and always verify from a machine outside the server rather than trusting ufw status.

Nothing obviously wrong, but who has been logging in?

last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"

Common mistakes

• Closing your only SSH session before testing the new configuration from a second one.
• Enabling the firewall before adding the SSH rule.
• Naming your sshd drop-in 99-, so a vendor file that sorts earlier wins.
• Trusting the file you edited instead of running sshd -T.
• Generating the SSH key on the server.
• Changing the SSH port and counting it as a security control.
• Disabling password auth for SSH while leaving a web control panel with a weak password exposed.
• No two-factor authentication on the hosting account that can reset root.
• Treating provider snapshots as backups.
• Installing fail2ban and never checking that it started.
• Publishing container ports without binding them to localhost, then assuming ufw covers it.
• Pasting a sysctl block you cannot explain line by line.

Best practices

• Verify every control after applying it, with a command that reads the running state.
• Keep a second session open and the provider console tested before making SSH changes.
• Key-only authentication, root login disabled, and an explicit AllowUsers list.
• Default-deny inbound firewall, with rules added before the firewall is enabled.
• Automatic security updates on, with a deliberate decision about automatic reboots.
• Two-factor authentication on the hosting account and the registrar.
• Backups off the machine, on separate credentials, and restore-tested at least once.
• UTC everywhere, with clock sync confirmed.
• Monitoring that pages you on disk usage before it pages you on downtime.
• Script the whole build, so the next server is identical and you are not repeating this from memory.

FAQ

What is the single most important step?

Key-only SSH with root login disabled, verified with sshd -T. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.

Should I change the SSH port?

For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.

Do I need fail2ban if password authentication is off?

Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.

Why did my sshd_config change get ignored?

Two likely causes. A file in /etc/ssh/sshd_config.d/ that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is Port or ListenAddress on a socket-activated system, where systemd owns the listener. sshd -T answers the first case and ss -tlnp answers the second.

Is ufw enough, or should I use nftables directly?

ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.

How do I harden a server I did not build?

Audit before you change anything. sshd -T, ufw status verbose, ss -tlnp for unexpected listeners, systemctl list-units --type=service --state=running, and the contents of every authorized_keys on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.

Should this be automated?

Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.

The one thing to remember

Editing a config file is not the same as changing a system’s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.

So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. sshd -T for SSH, ufw status verbose for the firewall, ss -tlnp for what is listening, systemctl status for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.

Want a second pair of eyes on your server?

Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:

• Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.
• Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.
• Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.
• Moving SSH off the public internet onto a private overlay network or bastion.
• Backup strategy that survives the hosting account being compromised, including a tested restore.
• Firewall and container networking, especially where published Docker ports are bypassing the firewall.

Send me the output of sshd -T, ufw status verbose and ss -tlnp, and I will tell you what stands out.

The post You Probably Didn’t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself appeared first on John Nessime.

Your server is only sending the leaf certificate. The intermediate that links it to a trusted root is missing. Your browser papers over that: it either has the intermediate cached from another site, or it quietly fetches it using the AIA extension in the certificate. curl, Java, Go, Python and every other non-browser client do no such thing. They see a certificate signed by something they do not trust and stop.

This is the most common of all TLS certificate errors and it is close to invisible from a desktop. The padlock in your browser is not a test. It is a client with a large safety net doing you a favour.

This post covers the four things a client actually checks and how each one fails: the chain, the names, the clock, and the trust store. Then the automation half, which has its own quiet failure: a renewal that succeeds and never reaches the running server. Plus the two things about TLS that have genuinely changed recently and will break config you copied from an older guide.

Look at what the server is actually sending

Before theorising, get the facts. One command tells you almost everything.

# -servername sends SNI. Without it you get whatever the default
# vhost serves, which is a different certificate on most servers
# and the source of a lot of confusing results.
# -showcerts prints every certificate the server sent, in order.
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

Read the top of that output. There is a Certificate chain block listing what arrived, numbered from zero, with s: for subject and i: for issuer. Entry 0 is your certificate. Entry 1 should be the intermediate that issued it, and its subject should exactly match entry 0’s issuer.

If the chain block has only entry 0, that is your bug. Right at the bottom, Verify return code gives the verdict: 0 (ok) is what you want, 21 (unable to verify the first certificate) means the chain is incomplete, and 20 (unable to get local issuer certificate) means the issuer is not in the local trust store.

Run it from a machine that is not yours. A clean container is ideal, because your laptop has accumulated intermediates and corporate roots that a customer’s server has not.

Chain problems

Almost all of these come down to one file choice. ACME clients write out several files and two of them look interchangeable:

• cert.pem is the leaf alone. Using this is the mistake.
• chain.pem is the intermediates alone.
• fullchain.pem is leaf plus intermediates, in the right order. This is the one your server should point at.

# nginx
ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

# Apache. On current versions the chain goes in SSLCertificateFile.
# SSLCertificateChainFile is deprecated; guides that still use it
# are old enough that other advice in them is probably stale too.
SSLCertificateFile      /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/example.com/privkey.pem

Order matters as well as contents. The leaf must come first, then each intermediate that signs the one above it. A chain assembled in the wrong order will be rejected by strict clients even though every certificate in it is correct. You can check a chain offline before deploying it:

# Does this leaf chain to a trusted root using these intermediates?
# Exits non-zero and explains itself if not.
openssl verify -untrusted chain.pem cert.pem

Do not include the root certificate in the file you serve. The client either has it or does not, and sending it wastes bytes on every handshake without changing any outcome.

Name problems

The second family. A certificate can be perfectly valid and still be the wrong certificate for the name being requested.

# The hostname the client asked for must appear in here.
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"

Three things people get wrong here:

• The Common Name does not count. Modern clients ignore the CN field entirely for hostname matching. If the name is only in the CN and not in the SAN list, it will fail, and the error message will not explain that clearly.
• Wildcards are exactly one label deep. *.example.com covers www.example.com and api.example.com. It does not cover example.com itself, and it does not cover a.b.example.com. The bare domain has to be listed separately, and it usually is not.
• SNI decides which certificate you get. One IP serving many sites picks the certificate based on the name the client sends during the handshake. A client that does not send SNI, or an openssl command without -servername, gets the default vhost’s certificate and produces a name mismatch that has nothing to do with your actual configuration.

Time and trust problems

# Local file
openssl x509 -in cert.pem -noout -dates

# Live endpoint, subject and dates in one go
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null 
  | openssl x509 -noout -dates -subject

Expired is obvious. notBefore in the future is the interesting one, because it almost never means the certificate is wrong. It means the client’s clock is wrong. Containers with no NTP, a VM restored from a snapshot, an embedded device with a dead battery: all of them produce “certificate not yet valid” against a perfectly good certificate. Check the clock on the failing machine before you touch anything on the server.

Trust store problems look similar but come from a different place. The root that signed your chain has to be in the trust store of the machine making the request, and old systems have old trust stores. A device that has not received an OS update in years may simply not know about a certificate authority that appeared since. There is no server-side fix for that beyond choosing a CA with broader legacy coverage, which is one of the few genuine arguments for a paid certificate.

The renewal that succeeded and changed nothing

Here is the second invisible failure, and it catches experienced people. Your renewal timer runs, the ACME client obtains a new certificate, writes it to disk, logs success. The site goes down anyway.

Nginx read the certificate into memory when it started. It does not watch the file. Neither does Apache, HAProxy, Postfix, or almost anything else. Until the service is reloaded it keeps serving the certificate it loaded at boot, which is the one that just expired. Everything in your logs says the renewal worked, because it did. The delivery step was never wired up.

Certbot handles this with deploy hooks, which run only when a certificate was actually renewed. Put a script in the hooks directory and it applies to every certificate on the box:

#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-services.sh
# Remember: chmod +x, or it silently never runs.
set -euo pipefail

# Validate the config BEFORE reloading. A reload with a broken
# config can take the site down at the exact moment you were
# trying to keep it up.
nginx -t
systemctl reload nginx

# Anything else that holds a copy of the certificate.
# systemctl reload postfix

Then prove the whole path works without waiting sixty days to find out:

# Exercises the full renewal against the staging environment.
# Installs nothing, counts against no rate limits, and will
# fail loudly if your challenge setup is broken.
certbot renew --dry-run

One caveat worth knowing: --dry-run does not run deploy hooks by default, precisely because nothing was deployed. So it validates the issuance path but not the reload. Test that separately by touching the service and confirming the served certificate changes.

Expiry automation, and why it is no longer optional

This part has a date attached, so it is worth stating plainly rather than dancing around it.

The CA/Browser Forum approved a phased reduction of maximum certificate lifetimes. The cap dropped from 398 days to 200 days on 15 March 2026, falls to 100 days on 15 March 2027, and reaches 47 days on 15 March 2029. Domain validation reuse periods shrink alongside it.

The practical consequence: any process that involves a person remembering to renew something once a year is already broken, and by the third phase you are renewing every six weeks. Whatever you were getting away with manually, stop. ACME with a timer is the baseline. Certbot is the default choice, acme.sh is lighter and easier to script around, and Caddy obtains and renews certificates with no configuration at all, which is a genuinely good reason to consider it for a simple reverse proxy.

Choosing a challenge type

• HTTP-01 proves control by serving a file over port 80. Simple, and it breaks whenever port 80 is closed, redirected wholesale to HTTPS in a way that mangles the challenge path, or sitting behind a proxy that does not forward it.
• DNS-01 proves control by writing a TXT record. More setup, needs an API token for your DNS provider, and it is the only option for wildcard certificates. It also works for hosts with no inbound internet access at all, which is why I reach for it on anything internal.

The trade-off with DNS-01 is a credential: your renewal process now holds a token that can edit your DNS zone. Scope it to the narrowest permission your provider offers, and if they support delegating just the _acme-challenge records to a separate zone, do that instead.

Monitor expiry externally

Do not rely on the renewal system reporting its own health. If it is broken, it is also probably not telling you. Let’s Encrypt has discontinued its expiration reminder emails, so if that was quietly acting as your safety net, it is not one any more.

Check from outside, against the live endpoint, the way a user would:

#!/usr/bin/env bash
set -euo pipefail

HOST="example.com"
DAYS=21

# -checkend takes a number of seconds and exits non-zero if the
# certificate expires within that window. No parsing dates by hand.
if ! openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null 
     | openssl x509 -noout -checkend $(( DAYS * 86400 )); then
  echo "TLS certificate for $HOST expires within $DAYS days" >&2
  exit 1
fi

If you already run Prometheus, the blackbox exporter gives you probe_ssl_earliest_cert_expiry for free, which is a timestamp you can alert on and graph in Grafana across every endpoint you own. That is the better answer at more than a handful of hosts, because it also catches the certificates nobody remembered existed.

Two pieces of config that are now wrong

OCSP stapling with Let’s Encrypt. Let’s Encrypt removed the OCSP responder URL from newly issued certificates in May 2025 and shut down its responders in August 2025. If your config still has this, it does nothing except print a warning on every reload:

# nginx: [warn] "ssl_stapling" ignored, no OCSP responder URL in the certificate
# Comment these out if your certificates come from Let's Encrypt.
# ssl_stapling on;
# ssl_stapling_verify on;

Nothing breaks by leaving them, but a warning that appears on every reload is a warning people learn to ignore, and then they ignore the next one too.

Hand-tuned cipher suite lists. If you pasted a ssl_ciphers line from a guide more than a couple of years old, re-check it against the current Mozilla SSL Configuration Generator. Lists that were considered hardened at the time now sometimes exclude things you want or include things you do not, and the defaults have improved considerably.

When there is a CDN in front

Behind Cloudflare or a similar edge, there are two separate TLS connections: browser to edge, and edge to your origin. They use different certificates and fail independently.

This creates a specific blind spot. Your origin certificate can be expired, self-signed or serving the wrong name, and visitors will see nothing wrong, because the edge presents its own valid certificate and, in the more permissive encryption modes, does not verify the origin. It works until you switch to strict origin validation, or until the edge is bypassed, and then everything fails at once for reasons that look unrelated to anything you changed.

Test the origin directly by name, bypassing DNS. If your origin is a VPS with a provider like InterServer and the public DNS points at the CDN, this is the only way to see what the origin is really serving:

# --resolve forces the hostname to a specific IP without touching
# /etc/hosts, so SNI and Host header stay correct.
curl -v --resolve example.com:443:203.0.113.10 https://example.com/

Troubleshooting by error message

unable to get local issuer certificate

Missing intermediate on the server, or a trust store on the client that lacks the root. Check the server first with -showcerts; if the chain is complete, the problem is on the client side.

certificate has expired, but I just renewed it

The service was never reloaded. Compare the dates on disk with the dates being served. If they differ, that is your answer, and the fix is a deploy hook rather than another manual renewal.

Hostname mismatch on a name that is definitely in the certificate

Either SNI is not reaching the right vhost, or the name is in the Common Name rather than the SAN list, or you are hitting a wildcard at the wrong depth. Check the SAN list and re-run s_client with an explicit -servername.

Works over IPv4, fails over IPv6 (or the reverse)

Two different machines are answering, and only one of them was updated. Force the family with curl -4 and curl -6 and compare. Load balancers with one member out of sync produce the same intermittent pattern.

Renewal fails with a challenge error

For HTTP-01, confirm that http://example.com/.well-known/acme-challenge/test is reachable from outside without being redirected or rewritten. A blanket redirect to HTTPS is the usual culprit. For DNS-01, check that the TXT record actually appears with dig, and give slow-propagating providers more time.

Everything looks right and it still fails

Run an external scan. SSL Labs or a local run of testssl.sh will check things you did not think to, including protocol support and chain issues from the perspective of clients you do not have installed.

Common mistakes

• Pointing the server at cert.pem instead of fullchain.pem.
• Treating a padlock in your own browser as proof the certificate is correctly deployed.
• Testing with openssl s_client without -servername and debugging the wrong certificate.
• Assuming *.example.com covers the bare domain.
• Putting the hostname only in the Common Name.
• Renewing without reloading the service.
• Forgetting chmod +x on a deploy hook, so it silently never runs.
• Relying on the renewal system to tell you when it is broken.
• Including the root certificate in the served chain.
• Leaving OCSP stapling enabled with certificates that no longer support it.
• Ignoring the origin certificate because the CDN hides its state.
• Debugging a “not yet valid” certificate on the server when the client’s clock is wrong.

Best practices

• Serve the full chain, leaf first, without the root.
• Verify from a clean machine or container, never from the laptop you browse on.
• Put every name you serve in the SAN list, including the bare domain alongside any wildcard.
• Automate renewal with ACME and a timer, and run certbot renew --dry-run after any change to the setup.
• Attach a deploy hook that validates the config before reloading, and make it executable.
• Monitor expiry externally against the live endpoint, and alert with weeks of headroom.
• Use DNS-01 for wildcards and internal hosts, with a narrowly scoped API token.
• Keep NTP running everywhere, including inside containers.
• Test the origin directly when there is a CDN in front, not just the public hostname.
• Inventory your certificates, because the one that takes you down is the one nobody remembered.

FAQ

Why does my certificate work in Chrome but fail in curl?

Your server is not sending the intermediate certificate. Browsers work around this by caching intermediates from other sites and by fetching them from the URL in the certificate’s AIA extension. Command-line tools and application HTTP libraries do neither. Switch the server to fullchain.pem and reload.

What is the difference between cert.pem, chain.pem and fullchain.pem?

Leaf only, intermediates only, and both concatenated in the correct order. Web servers want fullchain.pem. The private key stays in privkey.pem and never goes in any of them.

Does a wildcard certificate cover the root domain?

No. *.example.com matches exactly one label, so it covers www.example.com but not example.com and not a.b.example.com. Request the bare domain as an additional SAN, which most ACME clients will do if you pass both names.

How often should certificates renew now?

Whatever your CA issues, renew at roughly two-thirds of the lifetime so a failed attempt has room to retry before anything expires. Since maximum lifetimes are on a published downward schedule, build for automated renewal at any interval rather than tuning a number you will have to change again.

Should I still enable OCSP stapling?

Not with Let’s Encrypt certificates, which no longer carry a responder URL. If your CA still supports it, stapling remains worthwhile. Check whether your certificate actually contains an OCSP URL before enabling the directives, rather than assuming.

Is a paid certificate more secure than a free one?

Cryptographically, no. The encryption is identical. What money buys is broader compatibility with old trust stores, longer validation reuse, warranties, and human support when something goes wrong. For a normal public website, a free automated certificate is the right default.

How do I get a certificate for something with no public DNS or open ports?

DNS-01, on a public domain you control, pointed at whatever internal address you like. The CA only needs to see the TXT record; it never needs to reach the host. This is how internal services get publicly trusted certificates without being exposed.

The one thing to remember

Most TLS certificate errors are not about cryptography. They are about delivery: whether the server sends everything the client needs, whether the name it sends matches the name that was asked for, and whether the certificate sitting in memory is the same one sitting on disk.

So stop checking with your browser. Check with openssl s_client from a machine that owes you no favours, and make sure something outside your server is watching the expiry date. As lifetimes keep shrinking, the gap between a renewal process that works and one that merely appears to work gets narrower every year.

Certificates causing you trouble?

Certificate problems tend to arrive at bad moments and turn out to be one wrong file path. Things I help with:

• Diagnosing a certificate that works for some clients and not others, and telling you exactly which link in the chain is missing.
• Setting up ACME renewal with deploy hooks across nginx, Apache, HAProxy, mail servers and anything else holding a copy.
• Migrating to DNS-01 for wildcards and internal hosts, with properly scoped API credentials.
• Certificate inventory and expiry monitoring in Prometheus and Grafana, so nothing expires without warning.
• Preparing a certificate estate for shorter lifetimes, replacing anything that still depends on a person remembering.
• Origin and edge TLS configuration behind a CDN, including strict origin validation.

Send me the output of openssl s_client -connect yourhost:443 -servername yourhost -showcerts and the error your failing client reports, and I will tell you what is wrong.

The post It Works in My Browser: Debugging TLS Certificate Errors in Chains, SANs and Expiry appeared first on John Nessime.

That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer’s token, pushed a malicious commit, and then retroactively repointed every version tag at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner’s secrets into the build log. On public repositories, those logs are readable by anyone.

Nobody’s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.

A GitHub Actions VPS deployment is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.

Who can actually reach that key

Worth being explicit, because it is a longer list than people expect:

• Anyone with write access to the repository. A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.
• Every action in the job. Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.
• Anyone who can influence what runs. This is why pull_request_target combined with checking out the pull request’s own code is such a well-known foot-gun: it runs the fork’s code with the base repository’s secrets in scope.

The goal is not to make the key unreachable. It is to make a stolen key not worth much.

Decide how the runner authenticates

Three options, and the right one depends on what you already run.

1. A dedicated SSH key in repository secrets. The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.
2. An overlay network. Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.
3. A self-hosted runner on the server. No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.

One thing that sounds appealing and is not practical: allowlisting GitHub’s runner IP ranges at your firewall. The ranges are published through GitHub’s meta API, but they are enormous and they change, and every other GitHub customer’s job runs from inside them. It buys you very little for a lot of maintenance.

Narrow the key until it is boring

Assume the key leaks. What can the holder do? Work backwards from there.

Give it its own user. A deploy account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.

Restrict the key in authorized_keys. SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:

# ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions

The guard script decides what the key is allowed to do. Keep it short enough to read in one go:

#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" >&2
    exit 1
    ;;
esac

Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.

Scope the sudo entry too. The deploy user needs to reload a service, not run arbitrary commands as root:

# visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted deploy user with no sudo is a reasonable stopping point.

Verify the server, not just the key

Almost every tutorial on this topic contains StrictHostKeyChecking=no, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.

# Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com

Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.

The workflow

name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" > ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz > /dev/null

Three details in there worth calling out, because they are the ones people leave out.

-o BatchMode=yes stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.

--delete is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if .deployignore is wrong. Run it once with --dry-run against staging and read the list before you point it at production.

The final curl --fail is the difference between “the files copied” and “the site works”. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.

Third-party actions

There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain ssh and rsync for that reason.

Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.

Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.

Gate production properly

The environment: key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:

• Scoped secrets. The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.
• Required reviewers. The job pauses and waits for a named human to approve it. This is the single control that turns “anyone with write access can deploy” into “anyone with write access can request a deploy”.
• Deployment branch and tag rules. Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.

Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.

For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.

Troubleshooting

Permission denied (publickey)

The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses printf '%sn' rather than echo: it guarantees the newline regardless of what got stored.

After that, check permissions on the server: ~/.ssh must be 700 and authorized_keys 600, owned by the deploy user, and the home directory must not be group-writable. Add -vvv to the ssh command to see how far the handshake gets, and read journalctl -u ssh on the server for the reason, since the client is deliberately vague.

Host key verification failed

Either the known_hosts variable is empty, or the server’s host key genuinely changed. If you rebuilt the server, regenerate it with ssh-keyscan and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.

The job hangs until the timeout

SSH is waiting on a prompt nobody will answer. Add -o BatchMode=yes and it fails immediately with a readable reason instead.

rsync works from my laptop, fails from the runner

If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records $SSH_ORIGINAL_COMMAND, run the job once, and read what rsync actually sent.

Works on push, fails on pull requests

Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for pull_request_target to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.

A secret appeared in the logs

Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.

Common mistakes

• Using an existing personal SSH key instead of generating a dedicated one for the runner.
• Giving the deploy user full sudo, or a sudo rule with a wildcard in it.
• Disabling host key checking with StrictHostKeyChecking=no.
• Referencing third-party actions by tag in the job that holds your production key.
• Interpolating ${{ secrets.X }} directly into a run: script instead of passing it through env:.
• Leaving default write permissions on a job that only needs to read.
• Using cancel-in-progress: true on a deploy, so a second push interrupts the first mid-copy.
• Running rsync --delete at production without testing the exclude list.
• Treating “the files copied” as a successful deploy, with no health check.
• Deploying straight from branch pushes with no approval step.
• Relying on log masking to keep a leaked secret out of the output.
• Never rotating the deploy key, including after someone leaves the team.

Best practices

• Dedicated deploy user, dedicated key, used for nothing else.
• Restrict the key in authorized_keys with restrict and a forced command.
• Scope sudo to the exact commands the deploy needs.
• Pin every action to a full commit SHA and let Dependabot bump them.
• Keep the deploy job small: fewer steps means fewer things that can read the secret.
• Set permissions: explicitly, defaulting to read-only.
• Verify host keys from a stored known_hosts value.
• Use environments with required reviewers and deployment rules for production.
• Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.
• End every deploy with a real HTTP request against the public URL.
• Rotate the deploy key on a schedule and whenever someone leaves.

FAQ

Is it safe to store an SSH private key in GitHub secrets?

Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.

Should I use a ready-made SSH deploy action?

They work and they are convenient. My preference is plain ssh and rsync in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.

Can I use OIDC instead of a stored key?

Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub’s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.

Should I allowlist GitHub’s IP ranges on my firewall?

Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer’s runners share those ranges. Narrowing what the key can do achieves more.

Self-hosted runner or hosted runner?

Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.

How do I roll back a bad deploy?

Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.

How often should the deploy key be rotated?

On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to authorized_keys and updating the secret, so there is no good reason for it to be a project.

The one thing to remember

The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.

So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger’s code did with your credentials.

Want this built or reviewed?

Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:

• Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.
• Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.
• Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.
• Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.
• Pinning and auditing third-party actions, plus repository settings that limit what can run.
• Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.

Send me your workflow file and the output of ssh deploy@yourhost 'id', and I will tell you what a stolen key would get.

The post Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully appeared first on John Nessime.

That is the failure mode worth taking seriously. Not slowness, not downtime — the CDN cached a page that was personalised for one user and served it to everybody else. And you will not notice, because your browser holds a session and looks fine. The people seeing the broken version are the ones who never contact you.

Getting CDN setup right for a dynamic site is not about finding the right toggle in a dashboard. It is about deciding, per route, what is public and what is personal, and then making the origin say so clearly enough that no edge node has to guess.

Sort your routes before touching any config

Every URL on your site falls into one of three buckets. Write the list out — actually write it, this takes ten minutes and prevents the incident above.

• Static assets. CSS, JS, images, fonts. Identical for everyone, content-addressed if your build is sensible. Cache these hard, for a year.
• Public dynamic. Generated by the application but identical for every anonymous visitor. Blog posts, product pages, category listings. This is where the real performance win lives, and where most people leave everything uncached out of nervousness.
• Private dynamic. Anything personalised or authenticated. Carts, dashboards, checkout, admin, most API responses. Never cached at a shared cache, no exceptions.

The middle bucket is the whole game. If you cache only the first bucket you have built an asset CDN and left the expensive work untouched. If you accidentally cache the third, you have a data leak.

The header that separates the buckets

The single most useful thing to understand is the difference between private and no-store, because people use them interchangeably and they mean very different things.

# Static assets with a content hash in the filename
Cache-Control: public, max-age=31536000, immutable

# Public dynamic: edge caches it, browser revalidates
Cache-Control: public, s-maxage=300, max-age=0, must-revalidate

# Personalised: browser may cache, shared caches must not
Cache-Control: private, no-store

# Genuinely sensitive: nobody stores this anywhere
Cache-Control: no-store

private means “a browser cache may keep this, a shared cache may not”. no-store means “do not write this to disk anywhere”. For an authenticated dashboard you want both. For a bank statement you want no-store alone.

The other key pair is s-maxage versus max-age. s-maxage applies only to shared caches — the CDN — and overrides max-age there. That lets you cache a product page at the edge for five minutes while telling browsers to revalidate every time, so a price change propagates in five minutes rather than living in someone’s browser for an hour.

Set caching headers at the origin, per route, and let the CDN follow them. Configuring cache rules only in the CDN dashboard means the policy lives somewhere your application code cannot see and your staging environment does not have.

Vary, and how it quietly ruins your hit rate

Vary tells caches that the response depends on a request header, so a separate copy is stored per distinct value. Used precisely it is essential. Used carelessly it destroys your cache.

# Fine. A handful of encodings, a handful of variants.
Vary: Accept-Encoding

# Also fine if you genuinely serve different content per language
Vary: Accept-Encoding, Accept-Language

Now the one to avoid:

# Effectively disables caching. Every browser build is a distinct
# User-Agent string, so every visitor gets their own cache entry.
Vary: User-Agent

If you need device-specific responses, most CDNs expose a normalised device-type value with two or three possible values. Use that rather than the raw header.

Vary: Cookie deserves its own warning. It looks like a safe way to separate logged-in from anonymous traffic, and it is not — every visitor carries slightly different cookies from analytics and consent tools, so you get a near-unique cache key per person. The correct approach is to make the presence of a session cookie bypass the cache entirely, rather than varying on it.

Cookies are where sites actually break

Two rules, and between them they prevent most CDN incidents.

Rule one: a session cookie means bypass

If a request arrives carrying your session cookie, it should go to origin uncached. Most CDNs let you express this as a cache rule; in nginx, if you are running your own edge, it is a variable:

map $http_cookie $bypass_cache {
    default                 0;
    "~*wordpress_logged_in" 1;
    "~*woocommerce_items"   1;
    "~*comment_author"      1;
    "~*PHPSESSID"           1;
}

location / {
    proxy_cache            edge_cache;
    proxy_cache_bypass     $bypass_cache;   # do not serve from cache
    proxy_no_cache         $bypass_cache;   # do not write to cache
    proxy_cache_valid      200 301 302 5m;
    proxy_cache_use_stale  error timeout updating http_500 http_502 http_503 http_504;
    proxy_cache_lock       on;

    add_header X-Cache-Status $upstream_cache_status always;
    proxy_pass http://origin;
}

You need both directives. proxy_cache_bypass stops a logged-in user being served a cached anonymous page. proxy_no_cache stops their personalised response being written into the cache for everyone else. Setting only the first is a common and dangerous mistake — the read path is protected while the write path is wide open.

That X-Cache-Status header is not decoration. It is how you verify any of this actually works.

Rule two: strip cookies you do not need

Analytics and consent tools set cookies on every visitor. If your CDN treats any cookie as a bypass signal, your anonymous traffic stops being cacheable the moment you add a tag manager. Configure the CDN to ignore everything except the specific cookies that matter, and be explicit about which those are.

Query strings, and the cache-key explosion

By default many CDNs include the full query string in the cache key. Then a marketing campaign starts appending utm_source, utm_medium, fbclid and friends, and one page becomes thousands of cache entries that each miss exactly once.

Decide deliberately which parameters affect the response:

• Include parameters that change the content — ?page=2, ?sort=price, ?category=boots.
• Ignore tracking parameters entirely. They never change what the server renders.
• Normalise order where the CDN supports it, so ?a=1&b=2 and ?b=2&a=1 are one entry rather than two.

An allowlist is safer than a blocklist here. New tracking parameters appear constantly; the set of parameters your application actually reads changes rarely.

Keeping dynamic pages cached without going stale

The usual objection to caching public dynamic pages is that content changes. Two mechanisms solve this, and together they let you cache aggressively without serving stale content for long.

Stale-while-revalidate

Cache-Control: public, s-maxage=60, stale-while-revalidate=600, stale-if-error=86400

Read that as three separate promises. Fresh for sixty seconds. For the following ten minutes, serve the stale copy immediately while fetching a fresh one in the background — so nobody waits. And if the origin is throwing errors, keep serving the stale copy for a day rather than showing a 502.

That last directive turns your CDN into a partial outage shield. It is one of the highest-value headers on this page and it costs nothing.

Purge on change

Time-based expiry is a blunt instrument. If your application knows when a product price changed, tell the CDN immediately rather than waiting for a TTL.

Tag-based purging is the version worth building toward — tag responses by the entities they contain, then purge by tag when an entity changes:

# Origin response for a product page
Cache-Tag: product-1234, category-boots, layout-main

Update product 1234 and you purge one tag, invalidating the product page, every category listing it appears on, and the homepage if it is featured — without knowing any of those URLs in advance. Header name and support vary by provider, so check what yours calls it.

Verifying it before you point DNS

Test with curl, not a browser. Your browser has cookies, a warm local cache and an opinion; curl tells you the truth.

# What headers is the origin actually sending?
curl -sI https://example.com/products/boots | grep -i 'cache-control|vary|set-cookie'

# Twice in a row: second request should show a HIT
curl -sI https://example.com/products/boots | grep -i 'x-cache|age'
curl -sI https://example.com/products/boots | grep -i 'x-cache|age'

Now the test that matters most. Request an authenticated page with a real session cookie and confirm it never returns a cache hit:

curl -sI https://example.com/account 
  -H "Cookie: session=<a-real-session-value>" 
  | grep -i 'x-cache|cache-control'

You want BYPASS or MISS, and Cache-Control: private. If that ever shows HIT, stop and fix it before going further.

The leak test, which is the one people skip: fetch a personalised page with a session, then fetch the same URL with no cookies at all and confirm you get the anonymous version.

# With a session
curl -s https://example.com/account -H "Cookie: session=<value>" | grep -i 'welcome back'

# Immediately after, with nothing. Should NOT show the personalised content.
curl -s https://example.com/account | grep -i 'welcome back'

If the second command finds personalised content, the CDN cached a private response. That is the Monday-morning email, and you have just caught it before your customers did.

Test the edge before DNS points at it

curl -sI https://example.com/ --resolve example.com:443:<edge-ip>

That sends a real request to a specific edge node with the correct hostname and TLS SNI, without changing DNS. It is how you find out whether the whole thing works before any user does.

Things that break in ways nobody expects

Client IPs all become the CDN’s

Your logs fill with edge node addresses, rate limiting starts throttling everyone at once, and geolocation breaks. The real address arrives in a forwarded header, and your origin has to be configured to trust it — but only from the CDN’s address ranges. Trusting X-Forwarded-For from anywhere lets any visitor claim any IP they like, which turns a logging annoyance into an authentication bypass.

Redirect loops

The CDN talks to your origin over HTTP, the origin sees a non-HTTPS request and redirects to HTTPS, which comes back through the CDN, forever. Either terminate TLS at the origin too, or have the origin trust X-Forwarded-Proto when deciding whether to redirect.

POST responses getting cached

Shouldn’t happen by default, but a badly written cache rule can do it, and the result is one user’s form submission response served to the next person. Only cache GET and HEAD. Ever.

Websockets and streaming

Websocket upgrades and server-sent events need to pass through untouched. Buffering breaks streaming responses silently — the connection works, the data just arrives in one lump at the end, which looks like an application bug rather than a proxy setting.

Set-Cookie on a cacheable response

If a public page returns Set-Cookie and gets cached, every subsequent visitor receives that cookie — potentially somebody’s session identifier. Most CDNs refuse to cache responses carrying Set-Cookie, but not all, and not always. Check your origin is not setting cookies on pages you intend to cache.

Rolling it out without a bad week

1. Write down the route list. Three buckets, every path accounted for.
2. Set headers at the origin first, with the CDN not yet in front. Verify with curl.
3. Start with everything bypassed except static assets. A CDN that caches nothing still terminates TLS closer to users and is not a failure.
4. Add public dynamic routes one at a time, with short TTLs. Sixty seconds, then extend once you trust it.
5. Run the leak test after every rule change, not just at the end.
6. Watch origin request volume. It should drop noticeably. If it does not, something is bypassing that you think is caching.
7. Keep a purge command to hand and know how long a full purge takes before you need it.

Common mistakes

• Testing only in a logged-in browser, where cached-private-content bugs are invisible.
• Setting proxy_cache_bypass without proxy_no_cache, protecting reads but not writes.
• Vary: Cookie or Vary: User-Agent, which shatters the cache into near-unique entries.
• Leaving tracking parameters in the cache key.
• Configuring caching in the CDN dashboard only, so the policy is invisible to your code and absent from staging.
• Trusting forwarded IP headers from any source rather than only the CDN’s ranges.
• Using private where you needed no-store, or the reverse.
• Caching everything on day one and finding the problems in production.

Best practices

• Default to no caching, then allow specific routes. Opt-in beats opt-out when the failure mode is a data leak.
• Own the policy at the origin so it travels with your code and exists in staging.
• Use s-maxage for the edge and max-age for browsers. They are different problems.
• Always set stale-if-error. It is free outage insurance.
• Expose a cache status header and monitor hit ratio per route, not site-wide.
• Build purge-on-publish into the application rather than relying on TTLs alone.
• Make the leak test part of CI, so a future config change cannot reintroduce it quietly.

Frequently asked questions

Can a CDN cache dynamic pages at all?

Yes, and that is where most of the benefit is. Dynamic does not mean personalised. A blog post rendered from a database is identical for every anonymous visitor and can be cached at the edge for minutes. Only genuinely per-user content must bypass.

What is the difference between private and no-store?

private permits browser caching but forbids shared caches. no-store forbids writing the response to any cache at all. Use both for authenticated pages; use no-store alone for genuinely sensitive responses.

Why did my hit ratio collapse after adding a cookie banner?

Because the consent tool sets a cookie on every visitor and your CDN treats any cookie as a bypass signal, or you are varying on Cookie. Configure the CDN to ignore all cookies except the specific ones that indicate a session.

How do I verify private pages are not being cached?

Request the page with a real session cookie and confirm the cache status shows a bypass or miss, never a hit. Then request the same URL with no cookies and confirm the personalised content is absent. Do this after every cache rule change.

Should I cache API responses?

Public read-only endpoints, yes — with short TTLs and a cache key that includes whatever parameters affect the response. Anything requiring an Authorization header should be treated as private by default.

Do I still need caching at the origin?

Yes. The CDN protects you from repeat requests for the same URL; it does nothing for the first request, for uncacheable routes, or during a purge. Origin-level caching handles those.

Wrapping up

A CDN in front of a dynamic site is a classification problem wearing a performance costume. Sort your routes into public, private and static; express that at the origin with Cache-Control; make session cookies a hard bypass on both the read and write path; and keep tracking parameters out of the cache key.

Then test the way an anonymous visitor arrives, not the way you browse your own site. Two curl commands — one with a session, one without — catch the failure that matters, and they take about ten seconds. Run them every time you change a cache rule and the Monday email never arrives.

Want a CDN set up without the surprises?

I work with teams on edge caching for dynamic sites — route classification, cache headers that survive a redesign, purge-on-publish wired into the application, and the monitoring that tells you when hit ratio quietly drops.

If you would rather have this done carefully the first time, you can hire me on Upwork.

The post CDN Setup Without Breaking Dynamic Content appeared first on John Nessime.