Docker disk usage grows quietly because every part of it accumulates independently — images, stopped containers, volumes, build cache, and container logs, which are the one nobody expects and which no Docker command reports on. Cleaning it up is easy. Cleaning it up without destroying something takes about five more minutes.

This is the order I work through it: measure first, reclaim from safest to most destructive, then fix the thing that caused the growth so you are not back here next month.

Measure before you delete

Every cleanup starts here, and skipping it is how people delete the wrong thing:

docker system df

That gives you four rows — Images, Containers, Local Volumes, Build Cache — with a total size and a reclaimable figure for each. The reclaimable column is the one that matters. If build cache is eleven gigabytes and all of it is reclaimable, you have found your win and you never need to go near volumes.

For the itemised version:

docker system df -v

Now you can see individual images with their sizes, which containers are using what, and which volumes have no container attached. Read this before running anything destructive.

Two more views worth having open:

# What is actually running, and what is just sitting there stopped
docker ps -a --format 'table {{.Names}}t{{.Image}}t{{.Status}}t{{.Size}}'

# Images by age, oldest first
docker images --format 'table {{.Repository}}:{{.Tag}}t{{.Size}}t{{.CreatedSince}}'

Reclaim in order, safest first

These are ordered deliberately. Work down the list and stop as soon as you have enough space. Most of the time you never reach step four.

1. Stopped containers

# See what would go
docker ps -a --filter status=exited --filter status=created

docker container prune

This removes stopped containers and their writable layers. Anything running is untouched, and named volumes attached to those containers survive — only the container’s own layer goes.

The one caveat: if a stopped container held state in its writable layer rather than in a volume, that state is gone. That is a design problem rather than a cleanup problem, but it is worth a glance before you confirm.

2. Build cache

On any machine that builds images regularly, this is usually the biggest single win and it is completely safe. Build cache is derived data. Deleting it costs you a slower next build and nothing else.

# Everything older than a week
docker builder prune --filter "until=168h"

# Everything, no exceptions
docker builder prune -a

3. Dangling images

# Look first
docker images -f dangling=true

docker image prune

Without -a, this removes only dangling images — untagged layers left behind when a rebuild moved a tag to a new image. They are almost always genuine garbage.

4. Unused images — read this one carefully

docker image prune -a

Adding -a changes the meaning completely. It removes every image that does not have at least one container associated with it — running or stopped. So an image you pulled for a job that runs weekly, with no container currently existing, is gone. It will pull again, which is fine if you have bandwidth and the registry is reachable, and much less fine at 3am on a machine with a slow link.

Safer in most cases is an age filter, which keeps anything recent:

docker image prune -a --filter "until=336h"   # older than 14 days

5. Volumes — the one that loses data

Volumes are where databases live. Treat this step as a data operation, not a cleanup.

Before removing anything, look inside the candidates. This mounts each dangling volume read-only into a throwaway container so you can see what is actually in it:

docker volume ls -qf dangling=true | while read -r v; do
  echo "=== $v"
  docker run --rm -v "$v":/vol:ro alpine sh -c 'du -sh /vol; ls -la /vol' 2>/dev/null | head -12
done

If you see anything resembling a data directory, back it up before it goes:

docker run --rm 
  -v my-volume:/vol:ro 
  -v "$PWD":/backup 
  alpine tar czf /backup/my-volume.tar.gz -C /vol .

Only then:

docker volume prune

On recent Docker versions this prunes only anonymous volumes by default, with --all needed to include named ones — a genuinely good safety change. Older versions removed named volumes too. Check which behaviour you have before trusting it:

docker version --format '{{.Server.Version}}'
docker volume prune --help

The gigabytes Docker never tells you about

Here is the part that catches experienced people. Container logs do not appear in docker system df at all, and no prune command touches them. A chatty application with the default logging driver will happily write tens of gigabytes and nothing in Docker’s own tooling will mention it.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -10

To map a log file back to the container that owns it:

docker ps -aq | xargs docker inspect --format '{{.Name}} {{.LogPath}}'

Truncate, do not delete

This matters. Deleting a log file that a running container still has open removes the directory entry but not the data — the kernel keeps the blocks allocated until the process closes the descriptor, so df shows no improvement and you are left confused. Truncate instead:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Then stop it happening again

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

One thing people get wrong here: this applies to newly created containers only. Anything already running keeps the logging config it was created with. You have to recreate those containers for the limit to take effect — restarting them is not enough.

While you are in that file, cap the build cache too:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

Automating it without automating a disaster

A weekly cron is worth having, provided it only ever does the safe operations. The rule is simple: never put --volumes in anything unattended.

# /etc/cron.d/docker-prune
# Weekly, Sunday 03:00. Containers, networks, dangling images and build
# cache older than seven days. No volumes, ever.
0 3 * * 0 root /usr/bin/docker system prune -f --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Note what docker system prune covers without flags: stopped containers, unused networks, dangling images and build cache. It does not touch volumes unless you explicitly pass --volumes, and it does not remove tagged images unless you pass -a. Those two flags are the whole difference between a routine job and an incident.

If you are tempted to add --volumes to a cron job so you do not have to think about it again, you have swapped a disk space problem for a data loss problem on a timer.

Troubleshooting

Pruned everything, df has not moved

Deleted files still held open by a running process. Find them:

sudo lsof +L1 | head -20

Restarting the process that holds the descriptor releases the blocks. If it is the Docker daemon itself, a systemctl restart docker does it, at the cost of bouncing every container without a restart policy.

“No space left on device” but df shows free space

You are out of inodes rather than bytes. Docker’s layered storage creates enormous numbers of small files:

df -i

The fix is the same — prune images and build cache, which is where the file count lives.

overlay2 is huge and I have hardly any images

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2 2>/dev/null

Usually build cache or container writable layers rather than images. Do not delete anything under overlay2 by hand — the directory names are referenced in Docker’s metadata, and removing them directly corrupts the daemon’s view of the world. Always go through docker commands.

Running low and need space right now

Fastest safe sequence, in order:

docker builder prune -a -f
docker container prune -f
docker image prune -f
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Four commands, no volumes touched, no tagged images removed. That resolves the large majority of Docker disk usage emergencies.

Common mistakes

• Reaching for docker system prune -a --volumes first because it is the top search result.
• Putting --volumes in a cron job.
• Deleting log files instead of truncating them, then wondering why df did not change.
• Setting log limits in daemon.json and assuming existing containers pick them up. They do not.
• Removing directories under /var/lib/docker/overlay2 by hand.
• Storing application data in a container’s writable layer rather than a volume, so cleanup destroys it.
• Never running docker system df, and so deleting the wrong category entirely.
• Assuming inodes cannot be the problem.

Best practices

• Set log rotation in daemon.json on every host, on day one.
• Cap build cache with the builder GC settings rather than relying on manual pruning.
• Use named volumes for anything you care about, so cleanup and data are clearly separated.
• Use multi-stage builds and a real .dockerignore. Smaller images mean less to clean up.
• Automate only the safe operations, with an age filter, and log what the job removed.
• Alert on disk usage at 80% so cleanup is a scheduled task rather than an incident.
• On busy build hosts, give /var/lib/docker its own partition so a runaway cache cannot take the OS down with it.

Frequently asked questions

Is docker system prune safe to run?

Without flags, yes, on most systems. It removes stopped containers, unused networks, dangling images and build cache. Adding -a also removes tagged images with no container attached. Adding --volumes is the one that can lose data.

Will pruning delete my database?

Only if the database lives in a volume and you pass --volumes, or it lives in a container’s writable layer and you prune that container. If it is in a named volume attached to a running container, ordinary pruning will not touch it.

Why does docker system df not match du?

Two reasons. It does not count container logs at all, and it accounts for shared image layers once rather than per image. du -sh /var/lib/docker is the honest total.

How do I stop logs growing without limit?

Set max-size and max-file under log-opts in /etc/docker/daemon.json, restart the daemon, then recreate existing containers so they pick up the new configuration.

Can I move Docker’s storage to another disk?

Yes, with data-root in daemon.json. Stop Docker, copy /var/lib/docker across preserving ownership and extended attributes, point data-root at the new location, then start it again. Copy rather than move until you have confirmed it works.

Does pruning affect running containers?

No. Prune operations skip anything in use. The risk is always to things that are stopped, dangling or unattached — which is exactly why checking what is stopped before you prune is worth the thirty seconds.

Wrapping up

Docker disk usage is four categories that grow independently, plus a fifth — logs — that hides from Docker’s own accounting. Run docker system df, work down from build cache to volumes, and stop as soon as you have the space. The nuclear option exists, but reaching for it first is how people discover which of their volumes mattered.

Once the immediate problem is solved, spend ten minutes on log rotation and builder GC. Those two settings turn this from a recurring emergency into something you never think about again.

Need a hand with Docker in production?

I work with teams on container infrastructure — image sizes that got out of hand, build pipelines that cache badly, storage and logging defaults that were never set, and monitoring that catches a full disk before the alert does.

If you would rather not spend the evening on it, you can hire me on Upwork.

The post Cleaning Up Docker Disk Usage Safely appeared first on John Nessime.

For a single server, neither is the right answer. Prometheus and Grafana are two static Go binaries and a package. They run happily under systemd, use very little memory at this scale, and when something breaks you can read the logs and actually fix it. No container networking to debug, no orchestration layer between you and the problem.

This walks through the whole thing: install Prometheus and Grafana from scratch, wire in node_exporter, put the lot behind nginx with TLS, and lock down the ports that should never have been public in the first place. Commands work on both Debian-family and RHEL-family systems, with the differences called out.

What you are building

Four moving parts, all on one box:

• node_exporter exposes machine metrics — CPU, memory, disk, network — on port 9100.
• Prometheus scrapes that endpoint on a schedule and stores the samples in its own time series database on port 9090.
• Grafana queries Prometheus and draws the dashboards on port 3000.
• nginx terminates TLS and is the only thing listening on a public interface.

The critical design decision is in that last line. Prometheus and node_exporter ship with no authentication at all. Bound to a public interface, they hand your entire infrastructure inventory to anyone who runs a port scan. Everything binds to 127.0.0.1 and nginx is the single front door.

Resource-wise, a 2 GB VPS handles this comfortably for one server’s worth of metrics. Prometheus is the memory-hungry one, and its appetite scales with the number of active time series rather than the number of hosts.

Step 1: Users and directories

Neither service should run as root, and neither needs a login shell.

sudo useradd --system --no-create-home --shell /sbin/nologin prometheus
sudo useradd --system --no-create-home --shell /sbin/nologin node_exporter

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

On Debian and Ubuntu the nologin shell lives at /usr/sbin/nologin rather than /sbin/nologin. Check with which nologin if useradd complains.

Step 2: Install Prometheus

Prometheus ships as a single static binary. Rather than hardcoding a version that will be stale next month, pull the current release tag from the API:

PROM_VERSION=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')
echo "Installing Prometheus ${PROM_VERSION}"

cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"

sudo cp prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

If you are on ARM, swap linux-amd64 for linux-arm64.

The configuration file

Write /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
        labels:
          instance: "web-01"

Set that instance label deliberately. It follows every metric into every dashboard and alert, and renaming it later means your historical graphs split in two.

Validate before you start anything:

promtool check config /etc/prometheus/prometheus.yml

The systemd unit

Write /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/prometheus 
  --config.file=/etc/prometheus/prometheus.yml 
  --storage.tsdb.path=/var/lib/prometheus 
  --storage.tsdb.retention.time=30d 
  --storage.tsdb.retention.size=8GB 
  --web.listen-address=127.0.0.1:9090
ExecReload=/bin/kill -HUP $MAINPID

NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/prometheus
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Two things worth noticing. --web.listen-address=127.0.0.1:9090 is what keeps Prometheus off the public internet. And setting both retention flags matters — time-based retention alone will not save you if metric volume spikes, because Prometheus honours whichever limit it hits first.

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Step 3: node_exporter

Same pattern — a static binary and a unit file.

NE_VERSION=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest 
  | grep -Po '"tag_name": "vK[^"]*')

cd /tmp
curl -LO "https://github.com/prometheus/node_exporter/releases/download/v${NE_VERSION}/node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
tar xzf "node_exporter-${NE_VERSION}.linux-amd64.tar.gz"
sudo cp "node_exporter-${NE_VERSION}.linux-amd64/node_exporter" /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Then /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/node_exporter 
  --web.listen-address=127.0.0.1:9100

NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

# Confirm it is actually producing metrics
curl -s http://127.0.0.1:9100/metrics | head -20

Now open http://127.0.0.1:9090/targets — via an SSH tunnel, since it is not public — and both jobs should read UP.

# From your laptop
ssh -L 9090:127.0.0.1:9090 you@your-server

Step 4: Install Grafana

Grafana is packaged properly, so use the official repository rather than a tarball. It gets you signed updates through your normal patching process.

Debian and Ubuntu:

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key 
  | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" 
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install -y grafana

RHEL, AlmaLinux, Rocky:

sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana

Bind it to localhost too

Edit /etc/grafana/grafana.ini:

[server]
http_addr = 127.0.0.1
http_port = 3000
domain = metrics.example.com
root_url = https://metrics.example.com/

[security]
admin_user = admin
admin_password = put-something-real-here
cookie_secure = true

[users]
allow_sign_up = false

Getting root_url right matters more than it looks. Set it wrong and Grafana generates broken links and redirects behind the proxy, which produces a confusing half-working UI rather than a clean error.

sudo systemctl enable --now grafana-server

Step 5: Put nginx and TLS in front

This is the only piece that faces the internet.

server {
    listen 443 ssl;
    server_name metrics.example.com;

    ssl_certificate     /etc/letsencrypt/live/metrics.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/metrics.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Grafana Live needs websockets, and breaks silently without this
    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host       $host;
        proxy_pass http://127.0.0.1:3000;
    }
}

Note fullchain.pem rather than cert.pem. Getting that wrong produces a site that works in your browser and fails for anything else.

Then close the metrics ports at the firewall, belt and braces:

# firewalld
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

# ufw
sudo ufw allow 443/tcp

# Verify nothing unexpected is listening publicly
sudo ss -tlnp | grep -E '9090|9100|3000'

Every line of that last command should show 127.0.0.1. If you see 0.0.0.0 or *, something is exposed and you should fix it before going any further.

Step 6: Connect Grafana to Prometheus

You can click through the UI, but provisioning it as a file means a rebuilt server comes back identical. Create /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true

Restart Grafana, log in, and import dashboard ID 1860 from the Grafana dashboard library — Node Exporter Full. It is comprehensive, well maintained, and will show you more about your server than you knew was measurable.

Sizing the disk before it bites

Prometheus storage is predictable. Roughly:

disk = retention_seconds × samples_per_second × bytes_per_sample

Compressed samples land in the region of one to two bytes each. Measure your actual rate rather than guessing — run this in the Prometheus query box:

rate(prometheus_tsdb_head_samples_appended_total[5m])

One server with node_exporter at a 15 second interval is a small number, and 30 days fits in a few gigabytes. The number climbs fast once you add exporters, so keep --storage.tsdb.retention.size set as a hard ceiling. Prometheus filling the root partition takes the rest of the box down with it, which is a memorable way to discover your monitoring has no monitoring.

Troubleshooting

A target shows DOWN

Work outward from the exporter:

curl -s http://127.0.0.1:9100/metrics | head
sudo systemctl status node_exporter
sudo journalctl -u prometheus -n 50 --no-pager

On RHEL-family systems with SELinux enforcing, this is frequently SELinux rather than your config. Check before you start changing things:

sudo ausearch -m avc -ts recent

If that shows denials, write a policy for them. Do not reach for setenforce 0 — it fixes today’s symptom and quietly removes a layer of protection from the whole machine.

Grafana loads but every panel says “No data”

Almost always the datasource URL. It must be http://127.0.0.1:9090 from Grafana’s point of view, not your public hostname. Test the datasource from its settings page — Grafana reports the actual connection error, which is more useful than the empty panel.

Prometheus will not start

sudo journalctl -u prometheus -n 50 --no-pager
promtool check config /etc/prometheus/prometheus.yml
ls -la /var/lib/prometheus

The usual culprits are a YAML indentation error, or /var/lib/prometheus not being owned by the prometheus user after a manual file copy.

Broken styling or redirect loops behind the proxy

root_url in grafana.ini does not match how you are actually reaching the site. If you are serving from a subpath, you also need serve_from_sub_path = true.

Common mistakes

• Leaving 9090, 9100 or 3000 on a public interface. Prometheus has no authentication whatsoever.
• Keeping the default Grafana admin password because it prompts you to change it and you clicked past.
• Setting no retention limits and discovering the problem when the root partition fills.
• Dropping scrape_interval to 1s because more data seems better. It multiplies storage and buys you nothing at this scale.
• Never backing up /var/lib/grafana/grafana.db, which holds every dashboard you have built.
• Running everything as root because the tutorial you followed did.
• Building dashboards but configuring no alerts, so you still find out from a user.

Best practices

• Bind every component to localhost and expose exactly one TLS endpoint.
• Set both time and size retention. Whichever triggers first is the one you want.
• Provision datasources and dashboards as files, so the config lives in version control.
• Back up the Grafana SQLite database on the same schedule as everything else that matters.
• Add an external uptime check. Self-hosted monitoring cannot tell you that the server it lives on has gone.
• Write a handful of alerts you will genuinely act on. Alerts nobody responds to train people to ignore alerts.
• Use systemd’s sandboxing directives. They cost nothing and limit what a compromised exporter can reach.

Frequently asked questions

Should I use Docker instead?

For a single server, systemd is simpler. Two static binaries and a package, with no container networking layer between you and the logs. Containers earn their keep once you are managing several hosts or already run everything else that way.

How much RAM does this need?

A 2 GB VPS is comfortable for one server. Prometheus memory tracks active time series rather than host count, so it grows when you add exporters, not when you add machines with the same exporters.

Can I monitor more servers later?

Yes. Install node_exporter on each and add them as targets. The moment metrics cross a network, though, that traffic needs protecting — a VPN, an SSH tunnel, or TLS with client certificates. Do not simply open 9100 to the world.

Where do alerts come from?

Two options. Grafana’s built-in alerting works from dashboard queries and is the quicker path. Alertmanager, run alongside Prometheus, is more powerful for routing, grouping and silencing. Start with Grafana’s and move if you outgrow it.

How long should I keep metrics?

Thirty days covers most troubleshooting and capacity questions. Longer retention on a single box is usually a sign you want a remote-write backend rather than a bigger disk.

Do I need node_exporter if I only care about my application?

Install it anyway. When the application misbehaves, the first question is whether the machine underneath it is healthy, and node_exporter is what answers that.

Wrapping up

Once you install Prometheus and Grafana this way, the whole stack is four systemd units and three config files. Everything binds to localhost, nginx handles TLS, and there is no layer of abstraction between you and a problem when something breaks at an inconvenient hour.

Get the dashboard working first, then spend an evening writing three or four alerts you will actually respond to. A dashboard tells you what happened when you go looking. An alert is what stops you needing to look.

Want this set up properly?

I build monitoring stacks for teams who would rather not spend a weekend on it — Prometheus and Grafana on bare servers or in Kubernetes, exporters for the things you actually run, alert rules tuned so they mean something, and dashboards that answer real questions rather than filling a screen.

If that sounds useful, you can hire me on Upwork.

The post Prometheus + Grafana From Scratch on One Server appeared first on John Nessime.

Nearly every TLS certificate error I have had to chase down comes back to one of three things: the chain your server sends is incomplete, the certificate does not actually cover the hostname being requested, or it expired and no alert fired. This article walks through all three — how to identify which one you are looking at, the exact commands to prove it, and how to set up renewal automation so the third category stops being a category at all.

Why TLS certificate errors are so confusing to debug

Two things conspire to make this harder than it should be.

The first is that browsers are forgiving in ways other clients are not. When a server sends an incomplete certificate chain, most modern browsers will quietly fetch the missing intermediate using the Authority Information Access extension embedded in the certificate. They fix your misconfiguration on the fly and show you a padlock. Meanwhile curl, Python’s requests, Java clients, Go services, and older Android devices do no such thing. They fail. This is why “it works in Chrome” is worth almost nothing as a diagnostic signal.

The second is that error messages describe the symptom rather than the cause. “Unable to get local issuer certificate” sounds like a client problem. It usually is not — it usually means your server is not sending the full chain. Learning to translate the message into the actual failure is most of the work.

One command that tells you almost everything

Before touching config files, look at what the server is actually sending on the wire:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

The -servername flag matters more than people expect. It sets SNI, and without it you may be handed the server’s default certificate rather than the one for the domain you care about. Plenty of confusing “the certificate is for the wrong domain” reports come from testing without SNI.

Three parts of the output are worth reading carefully:

• The “Certificate chain” block near the top, listing each certificate the server sent with its subject and issuer.
• The “Verify return code” at the bottom. 0 (ok) is what you want. 20 and 21 both point at chain problems.
• The subject line of the first certificate, which tells you whether you even got the certificate you expected.

If you want the certificate details without the chain noise, pipe it into x509:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null 
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

That single line answers “who issued it”, “when does it expire”, and “what hostnames does it cover” in one shot. It is the first thing I run.

Family one: certificate chain issues

A publicly trusted certificate is almost never signed directly by a root. There is at least one intermediate CA in between. The client already trusts the root — it ships in the operating system or browser trust store — but it has no idea about the intermediate unless you hand it over. Your server’s job is to send the leaf certificate plus every intermediate needed to walk up to a trusted root.

The classic mistake: serving the leaf without the intermediates

If you use Let’s Encrypt, certbot writes several files into /etc/letsencrypt/live/example.com/. Two of them look interchangeable and are not:

• cert.pem — the leaf certificate on its own.
• fullchain.pem — the leaf plus the intermediates.
• chain.pem — the intermediates on their own.
• privkey.pem — the private key.

Nginx wants fullchain.pem. Point it at cert.pem and browsers will still work, so the mistake survives testing and surfaces weeks later when an API client fails.

server {
    listen 443 ssl;
    server_name example.com www.example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}

On Apache 2.4.8 and later, SSLCertificateFile can hold the leaf and the intermediates concatenated together, and SSLCertificateChainFile is deprecated. If you inherited a config that still uses the old directive, it will work, but it is worth cleaning up:

SSLCertificateFile    /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem

Wrong order, or a root that should not be there

Order is part of the specification, not a suggestion. The leaf comes first, then each issuer in sequence walking upward. Some clients tolerate a shuffled bundle; several do not, and the ones that do not tend to be the ones you find out about from an angry customer.

Including the root certificate at the end is a different case: it is harmless from a trust perspective, since the client ignores it and uses its own copy, but it adds bytes to every single handshake for no benefit. Leave it out.

Verifying a chain offline

You can check a bundle before deploying it, which is far better than discovering the problem in production:

# Does the leaf verify against the intermediates you plan to serve?
openssl verify -untrusted chain.pem cert.pem

# Print the subject and issuer of every certificate in a bundle, in order
openssl crl2pkcs7 -nocrl -certfile fullchain.pem 
  | openssl pkcs7 -print_certs -noout

That second command is the one I reach for most. Read the output top to bottom: the issuer of each certificate should be the subject of the next one down. The moment that pattern breaks, you have found your gap.

The key and the certificate do not match

Less common, but it produces a spectacularly unhelpful error. After a renewal where files were copied around by hand, confirm the pair still belongs together — the two hashes must be identical:

openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa  -noout -modulus -in privkey.pem | openssl md5

Family two: SAN mismatches

The Common Name field is dead. It has been deprecated for hostname matching for years, and Chrome stopped honouring it entirely back in 2017. What matters now is the Subject Alternative Name extension. If the hostname a client requested is not in the SAN list, the connection fails, regardless of what the CN says.

Check what a certificate actually covers:

# From a file
openssl x509 -noout -ext subjectAltName -in cert.pem

# From a live server
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null 
  | openssl x509 -noout -ext subjectAltName

The four SAN mistakes that keep recurring

• Apex without www, or www without apex. These are two distinct names. A certificate for example.com does not cover www.example.com. Both need to be in the SAN list, and both need to be passed to certbot with separate -d flags.
• Assuming a wildcard covers everything. It does not. *.example.com matches exactly one label. It covers api.example.com. It does not cover example.com itself, and it does not cover staging.api.example.com. Multi-level subdomains need their own wildcard or their own entry.
• A new subdomain that never made it into the renewal. You add metrics.example.com to nginx, point DNS at it, and forget that the certificate was issued for a fixed list of names. Adding a name means reissuing, not just reloading.
• Connecting by IP address. Almost no public certificate includes an IP SAN, so hitting the server by IP will always fail hostname verification. That is expected behaviour, not a broken certificate.

One more that catches people out: if you use Cloudflare in proxied mode, the certificate a visitor sees is Cloudflare’s edge certificate, not yours. Debugging the origin means bypassing the proxy and testing the origin IP directly with the Host header set correctly, or temporarily turning the orange cloud grey.

Family three: expiry, and why automation is no longer optional

Expiry used to be a calendar problem. You bought a certificate for a year, set a reminder, and dealt with it once. That era is closing.

In April 2025 the CA/Browser Forum passed ballot SC-081v3, which phases the maximum validity period for publicly trusted TLS certificates down from 398 days to 47 days. The schedule runs in three steps: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029. The first phase is already in force. Domain validation reuse windows shrink alongside it.

The 47-day target was chosen deliberately: short enough that manual renewal stops being viable at any meaningful scale. Automation is the point of the change, not a side effect of it.

If you run a handful of domains, this is a nuisance. If you run fifty, manual renewal is already finished as a strategy.

Getting ACME renewal right

Certbot installs a systemd timer or cron entry that attempts renewal twice a day and does nothing unless the certificate is close to expiry. Confirm it is actually there:

systemctl list-timers | grep certbot
certbot renew --dry-run

The dry run exercises the whole path against the staging environment without touching rate limits. Run it after any change to your web server config, not just after installing certbot. A renewal that worked last quarter can break because someone added a redirect that intercepts the ACME challenge path.

The trap: renewal succeeds, the site still serves the old certificate

This is the failure I see most often, and it is genuinely maddening because every log says success. Certbot writes the new certificate to disk. Nginx has the old one loaded in memory and keeps serving it until it is told otherwise. The certificate on disk is valid, the certificate on the wire is expired, and everything looks fine from the server’s point of view.

The fix is a deploy hook, which runs only when a certificate was actually renewed:

certbot renew --deploy-hook "systemctl reload nginx"

Better still, make it permanent by adding it to the renewal configuration at /etc/letsencrypt/renewal/example.com.conf, under the [renewalparams] section, so it survives however the renewal happens to be triggered:

renew_hook = systemctl reload nginx

Use reload rather than restart. A reload picks up the new certificate without dropping connections.

Remember that anything else terminating TLS needs the same treatment. A mail server, a database proxy, a Docker container with the certificate baked into its image — each one needs its own reload step, and containers in particular will happily run for months with an expired certificate mounted from the host.

Wildcards need DNS-01

Wildcard certificates cannot be issued through HTTP-01 validation. They require DNS-01, which means certbot needs to create a TXT record on your domain, which means API credentials for your DNS provider. Use a provider plugin rather than the manual mode — manual DNS-01 cannot be automated, and an unautomated wildcard on a 47-day cycle is an outage with a scheduled start time.

If issuance fails outright with a policy error, check your CAA records before assuming the ACME client is broken. A CAA record that names a different CA will block issuance entirely:

dig CAA example.com +short

Monitor from outside, not from the server

Checking the certificate file on disk tells you what should be served. Only an external check tells you what is being served. Given the reload trap above, that difference is the entire point.

A minimal check that works anywhere:

#!/usr/bin/env bash
# Usage: ./cert-days.sh example.com
# Exits non-zero if fewer than 21 days remain.
set -euo pipefail

DOMAIN="$1"
THRESHOLD=21

END=$(echo | openssl s_client -connect "${DOMAIN}:443" -servername "${DOMAIN}" 2>/dev/null 
      | openssl x509 -noout -enddate | cut -d= -f2)

DAYS=$(( ( $(date -d "${END}" +%s) - $(date +%s) ) / 86400 ))
echo "${DOMAIN}: ${DAYS} days remaining (expires ${END})"

[ "${DAYS}" -lt "${THRESHOLD}" ] && exit 1
exit 0

If you already run Prometheus, the blackbox exporter gives you this for free through the probe_ssl_earliest_cert_expiry metric. An alert rule looks like this:

groups:
  - name: tls
    rules:
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 21
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "TLS certificate expiring in under 21 days"

Set the threshold well above your renewal window. With 200-day certificates renewing at 30 days remaining, a 21-day alert gives you nine days of warning that something in the pipeline broke. Once 47-day certificates arrive, those numbers compress hard, and an alert that fires too late is the same as no alert.

A workflow for diagnosing TLS certificate errors

When something breaks, work through it in this order. It takes about a minute and rules out whole families of causes as you go.

1. Reproduce outside the browser. Use curl -v or openssl s_client. If the browser works and curl does not, you are almost certainly looking at a chain problem.
2. Read the verify return code. Code 20 or 21 means a chain issue. A hostname mismatch message means a SAN issue. A date error means expiry — or a wrong system clock on the client.
3. Print the dates. Rules expiry in or out in one command, and tells you whether the certificate on the wire matches the one on disk.
4. Print the SANs. Confirm the exact hostname being requested appears in the list. Watch for apex versus www.
5. Walk the chain. Each certificate’s issuer should be the next one’s subject. Find where the sequence breaks.
6. Compare disk against wire. If the file is newer than what is being served, you have a reload problem, not a certificate problem.
7. Check the client’s trust store. If everything above is clean and one specific client still fails, the problem has moved to their end — an old Android device, a Java service with its own cacerts, or a container image built on a base with a stale CA bundle.

Common mistakes worth avoiding

• Testing only in a browser, and mistaking its silent chain repair for a working configuration.
• Pointing the web server at cert.pem instead of fullchain.pem.
• Renewing without reloading the service that holds the certificate in memory.
• Running certbot renew on a schedule but never running --dry-run after config changes.
• Copying key and certificate files between servers by hand and losing track of which pair belongs together.
• Monitoring the file on disk rather than the certificate actually being served.
• Forgetting that mail servers, load balancers, and containers terminate TLS too, and need their own renewal path.
• Adding a subdomain to the web server config and assuming the existing certificate covers it.

Best practices that hold up over time

• Automate issuance and renewal from day one, even for a single domain. The habit matters more than the scale.
• Always use a deploy hook, and always reload rather than restart.
• Alert on days remaining, measured externally, with a threshold at least twice your renewal window.
• Keep an inventory of every hostname you hold a certificate for. You cannot monitor what you have forgotten about.
• Prefer DNS-01 with a provider plugin if you need wildcards, and never rely on manual DNS validation.
• Set CAA records deliberately so you know which CAs are permitted to issue for your domain.
• Test with a non-browser client before calling a certificate deployment finished.

Frequently asked questions

Why does my site work in Chrome but fail in curl?

Almost always an incomplete chain. Browsers fetch missing intermediate certificates automatically using the Authority Information Access extension; curl does not. Point your web server at the full chain file rather than the leaf certificate and both will work.

What does “unable to get local issuer certificate” actually mean?

The client could not build a path from your certificate to a root it trusts. Nine times out of ten the server is not sending its intermediates. Occasionally the client’s own CA bundle is out of date, which is common in older container base images.

Does a wildcard certificate cover the root domain?

No. *.example.com matches a single label, so it covers www.example.com but not example.com and not a.b.example.com. Include the apex explicitly as an additional SAN entry when you request the certificate.

My certificate renewed but the site still shows it as expired. Why?

The service holding the certificate has not reloaded. The new file is on disk; the old one is still in memory. Add a deploy hook that reloads the service on renewal, then verify from outside with openssl s_client rather than trusting the file timestamp.

How short will certificate lifetimes actually get?

Under the CA/Browser Forum schedule, 200 days applies from March 2026, 100 days from March 2027, and 47 days from March 2029. Private and internal CAs are not bound by these rules, but public certificates are.

Is the Common Name field still used at all?

Not for hostname validation. Modern clients read only the Subject Alternative Name extension. A certificate whose CN matches but whose SAN list does not include the hostname will be rejected.

Wrapping up

TLS certificate errors feel opaque mostly because the error text points at the wrong layer. Once you sort them into chain, SAN, or expiry, each one has a short diagnostic path and an obvious fix. Get comfortable with openssl s_client and openssl x509 and you will resolve most incidents faster than it takes to find the right dashboard.

The expiry category is the one worth engineering away permanently. With maximum lifetimes dropping toward 47 days, any process that depends on a person remembering something will fail eventually. Automate the renewal, add the deploy hook, and alert on what is actually being served rather than what is sitting on disk. Do that once and TLS certificate errors go back to being a rare curiosity instead of a recurring outage.

Need help with TLS, certificates, or infrastructure automation?

I work with teams on exactly this kind of problem — broken certificate chains, ACME automation across multiple servers, expiry monitoring wired into Prometheus and Grafana, and the wider DevOps plumbing that keeps it all running quietly in the background.

If you would rather hand this off than debug it at two in the morning, you can hire me on Upwork.

The post TLS Certificate Errors: Chain Issues, SANs, and Expiry Automation appeared first on John Nessime.

It usually is. Not because DNS is fragile, but because it’s the one layer where caching, delegation, and propagation all conspire to make the same domain behave differently depending on who’s asking and when. A record can be perfectly correct in your control panel and still resolve to the wrong IP for half the internet.

This article is the DNS troubleshooting workflow I actually use — the order of checks, the commands worth memorising, how to read what they tell you, and the specific failure patterns that show up over and over. No theory dumps. Just the path from “something’s broken” to “here’s exactly which record on which server is wrong.”

First, Get the Mental Model Right

You can’t debug a system you don’t have a picture of. When a browser resolves example.com, several independent parties are involved, and any one of them can be the liar:

1. The application — browsers keep their own short-lived DNS cache, and HSTS or connection pooling can mask changes entirely.
2. The operating system — the local stub resolver, /etc/hosts, nscd, systemd-resolved, or the Windows DNS Client service.
3. The recursive resolver — your router, your ISP, or a public resolver like 1.1.1.1 or 8.8.8.8. This is where most stale answers live.
4. The delegation chain — root servers, then the TLD servers, then whichever name servers your registrar says are authoritative.
5. The authoritative name server — the only place where the truth actually lives.

Effective DNS troubleshooting is just working down that list and asking each party the same question directly, instead of trusting whatever the browser tells you. The moment you find two parties giving different answers, you’ve found your boundary — and the problem is between them.

The Tools That Matter

You need surprisingly few. My honest ranking:

dig — the one you should learn properly

dig is the only tool that shows you everything: the query it sent, the response status, the TTL remaining, the authority section, and whether the answer came from a cache or from the source. On Debian and Ubuntu it lives in dnsutils; on RHEL-family systems it’s in bind-utils.

# Just the answer, nothing else
dig +short example.com A

# The full picture — status, flags, TTL, authority
dig example.com A

# Ask one specific resolver instead of the system default
dig @1.1.1.1 example.com A

# Ask a specific authoritative server directly
dig @ns1.example-dns.com example.com A

host — for quick sanity checks

host example.com
host -t MX example.com
host -t TXT example.com

nslookup — fine, but know its limits

nslookup is everywhere, including Windows, which is its main virtue. But it hides details you often need, and its “Non-authoritative answer” banner confuses people constantly — it simply means the answer came from a cache rather than the authoritative server. That’s completely normal. On Windows I reach for PowerShell instead:

Resolve-DnsName example.com -Type A
Resolve-DnsName example.com -Server 1.1.1.1

resolvectl — essential on modern Linux

If the machine runs systemd-resolved, your /etc/resolv.conf probably points at 127.0.0.53, which tells you nothing about the real upstream servers. resolvectl does:

resolvectl status
resolvectl query example.com
resolvectl statistics

getent — the reality check

dig talks to DNS. Your applications talk to the C library resolver, which also consults /etc/hosts and whatever else /etc/nsswitch.conf lists. When dig and the application disagree, this is the command that explains why:

getent hosts example.com
getent ahosts example.com

A Repeatable DNS Troubleshooting Workflow

This is the sequence. Don’t skip ahead — each step eliminates a whole class of causes, and jumping straight to the authoritative server is how you spend an hour fixing something that was never broken.

Step 1 — Confirm it’s actually DNS

Resolve the name, then connect to the resulting IP directly. If the IP works and the name doesn’t, it’s DNS. If neither works, you’re troubleshooting the wrong layer.

dig +short example.com A
curl -I --resolve example.com:443:203.0.113.10 https://example.com/

The --resolve flag is underrated. It forces curl to use an IP you specify while still sending the correct Host header and SNI, which lets you test the web server as if DNS were already fixed. Replace 203.0.113.10 with the IP you expect.

Step 2 — Compare local vs public resolvers

dig +short example.com A
dig +short @1.1.1.1 example.com A
dig +short @8.8.8.8 example.com A
dig +short @9.9.9.9 example.com A

If your local answer differs from all three public resolvers, the problem is on your side — a cache, a hosts file entry, a VPN pushing internal DNS, or a router doing something creative. If all four agree but the answer is wrong, the problem is upstream at the authoritative layer.

Step 3 — Find out who is actually authoritative

This is the step people skip, and it’s the one that catches the nastiest problems. The name servers listed in your DNS provider’s dashboard are not necessarily the ones the internet is using.

# What the parent zone (the registry) says
dig +short NS example.com

# What the zone itself says, straight from one of those servers
dig @ns1.example-dns.com example.com NS

Those two lists should match. When they don’t, you’ve usually found a half-finished DNS migration: the registrar still delegates to the old provider, so the world keeps reading a zone nobody has updated in months.

Step 4 — Query every authoritative server individually

A zone with four name servers can be inconsistent across them if a zone transfer failed. Intermittent, “it works every third time” problems are almost always this. Compare the SOA serial on each:

# Pull the SOA record from every authoritative server in one shot
dig +nssearch example.com

# Or check them one by one
dig +short @ns1.example-dns.com example.com SOA
dig +short @ns2.example-dns.com example.com SOA

Different serial numbers mean the secondaries haven’t caught up with the primary. Different answers for the same record mean something is genuinely broken in replication.

Step 5 — Walk the delegation chain

When nothing else explains it, follow the query the way a recursive resolver would — from the root, down through the TLD, to the authoritative servers:

dig +trace example.com A

Read it top to bottom. Each block is one hop. The point where the output stops making sense — a referral to servers that don’t answer, a missing glue record, an unexpected NXDOMAIN — is your break point. This single command has saved me more time than any other in DNS troubleshooting.

Reading dig Output Without Guessing

The status line in a dig response tells you the category of failure immediately. Learn these four and you’ve eliminated most of the guesswork.

• NOERROR — the query succeeded. Note that NOERROR with an empty answer section is not a failure: it means the name exists but has no record of the type you asked for. Asking for an A record on a name that only has MX records gives you exactly this.
• NXDOMAIN — the name does not exist at all. Not a typo in the record; the label itself isn’t in the zone. Check for a missing record, a misspelled hostname, or a zone that isn’t loaded.
• SERVFAIL — the resolver tried and failed. This is the ambiguous one, and in practice it usually means DNSSEC validation failure, an authoritative server that’s unreachable or refusing queries, or a broken delegation.
• REFUSED — the server you asked is not willing to answer for that zone. Typically you’re querying a server that isn’t authoritative and doesn’t do recursion for you.

Two other things worth reading in every response:

• The aa flag — present when the answer came from an authoritative server. If it’s missing, you’re looking at a cached answer, and the TTL tells you how stale it might be.
• The TTL column — on a cached answer this counts down. Query the same resolver twice a few seconds apart; if the TTL drops, you’re being served from cache. If it stays constant, you’re hitting the source.

The Propagation Myth (and What’s Really Happening)

“DNS propagation takes 24 to 48 hours” is the most repeated and least accurate sentence in web hosting. Nothing propagates. When you change a record, the authoritative server has the new value instantly. What takes time is the expiry of cached copies held by thousands of independent resolvers — and that duration is entirely determined by the TTL you set before the change.

The practical consequence: lowering a TTL after you’ve made the change does nothing for anyone who already cached the old value. Plan ahead instead.

1. Well before a migration, drop the TTL on the records you’ll change to something short — 300 seconds is a common choice.
2. Wait at least the length of the old TTL so the short value fully replaces it in caches.
3. Make the actual change. Old cached answers now expire within minutes.
4. Verify against several public resolvers and from a couple of different networks.
5. Once everything is stable, raise the TTL back to something sane — an hour or more — so you’re not paying for lookups you don’t need.

The TTL you set today determines how painful next month’s migration will be. It’s the cheapest piece of planning in infrastructure work, and the one most consistently skipped.

One more thing people miss: negative caching. When a lookup returns NXDOMAIN, resolvers cache that “doesn’t exist” answer too, for a duration derived from the minimum field in the zone’s SOA record. So if you query a hostname before creating it, you may keep getting NXDOMAIN for a while even after the record is live. Check what you’re in for:

dig +short example.com SOA

The last number in that output is the negative caching TTL. Resist the urge to test a hostname before you’ve actually created it.

Failure Patterns You’ll Meet Again and Again

A CNAME on the zone apex

You cannot put a plain CNAME on the root of a domain. The apex must carry SOA and NS records, and a CNAME is not allowed to coexist with other records at the same name. Providers work around this with ALIAS, ANAME, or “CNAME flattening” records that resolve the target server-side and return an A record. If your root domain misbehaves while www is fine, this is the first thing to check.

Stale cache — everywhere at once

Correct at the authoritative server, wrong on your machine. Clear caches from the outside in:

# Linux with systemd-resolved
sudo resolvectl flush-caches

# Linux with nscd
sudo systemctl restart nscd

# macOS
sudo dscacheutil -flushcache
sudo killall -HUP mDNSResponder

# Windows (run as Administrator)
ipconfig /flushdns

Then remember the browser. Chromium-based browsers keep a separate cache you can clear at chrome://net-internals/#dns. And if the site uses HSTS, a browser may refuse to fall back to HTTP regardless of what DNS says — test with curl before you conclude anything from a browser.

A forgotten line in /etc/hosts

Someone adds a hosts entry during a migration to test the new server, and never removes it. Six months later the server is “randomly” hitting the wrong backend. It costs three seconds to rule out:

grep -i example.com /etc/hosts
getent hosts example.com

DNSSEC validation failures

A domain with broken DNSSEC returns SERVFAIL from validating resolvers while working perfectly on non-validating ones. That produces the maddening symptom where the site loads for some users and not others, with no geographic pattern.

The diagnostic trick is +cd, which disables validation. If the query fails normally but succeeds with +cd, DNSSEC is your culprit:

dig @1.1.1.1 example.com A
dig @1.1.1.1 +cd example.com A

# Show the DNSSEC records themselves
dig example.com DNSKEY +dnssec
dig example.com DS

# delv performs validation and explains the outcome
delv example.com A

The usual cause is a DS record at the registrar that no longer matches the keys published in the zone — classic after changing DNS providers without disabling DNSSEC first. Always turn DNSSEC off at the registrar before migrating name servers, then re-enable it once the new provider is live and stable.

Split-horizon DNS surprises

Internal resolvers hand out private addresses; external resolvers hand out public ones. Perfectly intentional — until a VPN, a container, or a misconfigured search domain puts a client on the wrong side of the split. When a name resolves differently depending on where you run the query, compare the local resolver against a public one and check whether a VPN profile is pushing its own DNS servers.

Proxied records hiding the origin

If a domain sits behind Cloudflare or a similar proxy, public DNS returns the proxy’s IP, not your server’s. That’s the whole point — but it means you cannot verify an origin change through public DNS at all. Query the authoritative provider’s own API or dashboard for the real record value, and test the origin directly with curl --resolve.

Wildcards masking a missing record

A * record in the zone means every hostname resolves — including the ones you typo’d, and including subdomains you thought you’d deleted. If everything resolves but half of it points somewhere unexpected, check for a wildcard before you go hunting for phantom records.

Missing trailing dots in zone files

Hand-editing BIND zone files? A record value without a trailing dot gets the origin appended to it. Write mail.example.com instead of mail.example.com. and you’ve just created mail.example.com.example.com. Validate before reloading:

named-checkconf
named-checkzone example.com /var/named/example.com.zone

And bump the SOA serial. A zone edit without a serial increment won’t reach the secondaries, which is exactly the inconsistency you were chasing in Step 4.

Containers and the ndots trap

Inside Kubernetes pods, /etc/resolv.conf commonly sets ndots:5. Any name with fewer than five dots gets tried against every search domain first, so an external lookup like api.example.com can generate several failed queries before the correct one. It works — it’s just slow, and under load it looks like intermittent DNS failure. Adding a trailing dot to fully qualify the name skips the search list entirely:

cat /etc/resolv.conf
dig api.example.com.        # note the trailing dot

DNS Troubleshooting for Email Delivery

Mail problems are DNS problems more often than they are mail-server problems. When messages bounce or land in spam, check these before touching the mail server config:

# Where should mail for this domain go?
dig +short example.com MX

# SPF and DMARC live in TXT records
dig +short example.com TXT
dig +short _dmarc.example.com TXT

# DKIM lives under a selector you choose
dig +short selector1._domainkey.example.com TXT

# Reverse DNS for the sending IP
dig -x 203.0.113.10 +short

Things that regularly go wrong here:

• Two SPF records. A domain must publish exactly one. Two SPF TXT records is a permanent error, and receiving servers are entitled to fail the check outright. Merge them into one.
• An MX pointing at a CNAME. MX targets must resolve to address records, not aliases.
• Missing reverse DNS. If the PTR record for your sending IP doesn’t match the hostname the server announces, expect a lot of spam folder.
• DKIM selector mismatch. The selector in the message header must match the one published in DNS — a detail that quietly breaks after a mail platform migration.

While you’re in TXT territory: if certificate issuance is failing, check for a CAA record. A CAA entry that only authorises one certificate authority will silently block every other one, and the error you get back from the ACME client rarely says “DNS.”

dig +short example.com CAA

Mistakes That Cost the Most Time

• Testing only in a browser. Browser caches, HSTS, service workers, and QUIC connection reuse will all lie to you. Confirm with dig and curl.
• Trusting the control panel. The dashboard shows what the zone should contain. Only a query against the authoritative server shows what it actually serves.
• Changing several things at once. Adjust one record, verify, then move on. Batch changes turn a five-minute fix into an archaeology project.
• Forgetting IPv6. A stale or wrong AAAA record produces “works for some people” behaviour, because dual-stack clients generally prefer IPv6. Always check dig +short example.com AAAA.
• Leaving DNSSEC enabled through a provider migration. The single most reliable way to take a domain fully offline for hours.
• Not documenting the zone before a migration. Export or record every record first. Rebuilding a zone from memory at 2am is not a plan.

Best Practices That Prevent the Next Incident

• Use at least two name servers, ideally on separate networks or providers. Single-provider DNS is a single point of failure for your entire presence.
• Keep TTLs deliberate. Long TTLs for stable infrastructure, short ones ahead of planned changes. Not the other way round.
• Monitor DNS resolution itself, not just HTTP endpoints. A check that queries your authoritative servers directly and alerts on a changed or missing answer catches problems before users do.
• Version-control your zone data if your provider supports API or zone-file export. Diffing two zone versions turns “what changed?” into a one-second question.
• Keep a record inventory — what each record is for, who owns it, and whether it’s still needed. Old A records pointing at decommissioned servers are a genuine security risk, not just clutter.
• Standardise your checks. A short script that runs your Step 1–5 sequence against a domain makes DNS troubleshooting repeatable instead of improvised.

A Quick Reference You Can Keep

# Basic resolution
dig +short example.com A
dig +short example.com AAAA

# Compare resolvers
dig +short @1.1.1.1 example.com A
dig +short @8.8.8.8 example.com A

# Authority and delegation
dig +short example.com NS
dig +trace example.com A
dig +nssearch example.com

# Cache behaviour
dig example.com A            # watch the TTL count down
dig +norecurse @1.1.1.1 example.com A   # cached-only lookup

# DNSSEC
dig +cd example.com A
delv example.com A

# Mail and verification records
dig +short example.com MX
dig +short example.com TXT
dig +short _dmarc.example.com TXT
dig -x 203.0.113.10 +short

# System-level view
resolvectl status
getent hosts example.com
cat /etc/resolv.conf

Frequently Asked Questions

How long does a DNS change really take to take effect?

At the authoritative server, immediately. For everyone else, up to the TTL that was in place when they last cached the record. If the old TTL was 3600 seconds, worst case is roughly an hour. The “24 to 48 hours” figure is a legacy of very long default TTLs and badly behaved resolvers, not a rule.

What’s the difference between SERVFAIL and NXDOMAIN?

NXDOMAIN is a definitive answer: the name doesn’t exist. SERVFAIL means the resolver couldn’t produce an answer at all — commonly a DNSSEC validation failure, an unreachable authoritative server, or a broken delegation. NXDOMAIN points you at the zone contents; SERVFAIL points you at the infrastructure.

Why does the site work for me but not for a colleague?

Different resolvers with different cache states, a hosts file entry on one machine, a VPN pushing internal DNS, an IPv6 versus IPv4 difference, or DNSSEC failing only on validating resolvers. Have both people run the same dig against the same public resolver — if the answers match, the difference is local to one machine.

Should I use dig or nslookup?

dig, whenever it’s available. It shows response status, flags, TTLs and the authority section, all of which you need for real DNS troubleshooting. Use nslookup or Resolve-DnsName when you’re on a Windows box without other tooling.

Do online “DNS propagation checker” sites actually help?

They’re useful for one thing: seeing what different resolvers around the world currently have cached. They won’t tell you why an answer is wrong, and they can’t see proxied origins or internal split-horizon zones. Treat them as a rough sanity check, not a diagnosis.

Can a CNAME point to another CNAME?

Technically yes, and resolvers will follow the chain — but every extra hop adds latency and another thing to break. Keep chains short, and never let one loop back on itself. Also remember a CNAME cannot coexist with other records at the same name, which is why it doesn’t belong on a zone apex.

Why do my DNS changes keep reverting?

Usually because something else owns the zone: an infrastructure-as-code pipeline, a control panel that rewrites records on certain actions, or an external DNS controller in a Kubernetes cluster. Decide on one source of truth and make every change through it.

Is switching to a public resolver a fix?

It’s a workaround for one user, not a fix for the domain. It’s genuinely useful as a diagnostic — if 1.1.1.1 gives the right answer and your ISP’s resolver doesn’t, you’ve isolated the problem to that resolver’s cache. But if the record itself is wrong, changing resolvers just changes who tells you the wrong thing.

Wrapping Up

Good DNS troubleshooting isn’t about knowing obscure record types. It’s about discipline: confirm it’s DNS, compare resolvers, find out who’s really authoritative, ask each of them directly, and follow the delegation chain when the answers stop agreeing. Five steps, in order, every time.

Do that consistently and most incidents collapse into something small and obvious — a stale cache, a TTL nobody lowered, a DS record left behind after a migration, a hosts entry from six months ago. The problems aren’t usually exotic. They’re just invisible until you ask the right party the right question.

Keep the quick-reference commands somewhere you can reach in thirty seconds, set your TTLs before you need them, and monitor resolution as seriously as you monitor uptime. Future you will appreciate it at 2am.

Need a Hand With DNS, Hosting or Cloud Infrastructure?

If you’re stuck on a DNS issue, planning a migration you’d rather not do twice, or you just want someone to review a zone before you flip the switch, I’m available for freelance work.

I work on DNS configuration and troubleshooting, domain and hosting migrations, SSL and certificate issues, email deliverability records (SPF, DKIM, DMARC), server setup and hardening, AWS and cloud infrastructure, and monitoring and alerting — plus WordPress performance and maintenance.

You can see my profile, past work and reviews on Upwork, and message me directly there with what you’re dealing with.

Got a DNS problem this article didn’t cover? Leave a comment describing the symptoms and what dig +trace shows — that output alone usually narrows it down fast.

The post DNS Troubleshooting: How to Find the Real Cause in Minutes, Not Hours appeared first on John Nessime.