The line you wrote was real. The file was saved. What you did not know is that your distro puts an Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, that sshd uses the first value it finds for any given keyword, and that your cloud provider’s image dropped a 50-cloud-init.conf in that directory setting PasswordAuthentication yes. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.

You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.

That is the thing most guides get wrong, and it is the idea this VPS hardening checklist is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.

This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.

Before you touch anything

Two minutes of preparation that saves an evening.

Find your provider’s console before you need it. Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.

Never close the session you are working in. Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.

Then update the system, because everything else assumes current packages:

sudo apt update && sudo apt upgrade -y
sudo reboot   # if a kernel update landed

1. A real user, and keys instead of passwords

Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.

# On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"

# On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10

Now open a second terminal and confirm ssh deploy@203.0.113.10 works, and that sudo -v works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.

2. SSH config, and why your edit might do nothing

This is the section that actually matters. Two mechanisms silently override what you write.

Drop-in files and first-match-wins

Modern OpenSSH packages ship an Include line near the top of /etc/ssh/sshd_config, which pulls in every .conf file from /etc/ssh/sshd_config.d/ in alphabetical order, before the rest of the main file is read. Combine that with sshd’s rule that the first value found for a keyword wins, and the consequence is specific: your drop-in has to sort earlier than the vendor’s, not later.

Most guides tell you to name it 99-something.conf, which is the convention almost everywhere else in Linux and exactly backwards here.

# /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy

A note on KbdInteractiveAuthentication: older guides and older OpenSSH call this ChallengeResponseAuthentication. Setting only PasswordAuthentication no can still leave a keyboard-interactive path open on some configurations, which is why both are here.

Now the part nobody does. Validate the syntax, then read the effective config:

# Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'

If passwordauthentication yes comes back, your file lost to something else in that directory. Look at what is in there with ls /etc/ssh/sshd_config.d/ and either rename yours to sort earlier or edit the offending file directly.

Socket activation, and the port that will not change

The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: ssh.socket holds the listening port and starts sshd only when a connection arrives. On those systems the Port and ListenAddress directives in sshd_config may be ignored entirely, because systemd owns the socket.

People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:

# -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp

If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under /lib/systemd/system/, which a package update will overwrite:

sudo systemctl edit ssh.socket

# In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it

Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.

3. Firewall, in the right order

The ordering here is the whole trick. Set the default policy and add your rules before you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose

Read that status verbose output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.

One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with -p 5432:5432 is reachable from the internet even though ufw says that port is denied. Bind container ports to 127.0.0.1 explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.

A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.

4. Automatic security updates

Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug

By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.

5. fail2ban, and what it is actually for

Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.

What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials do exist: a mail server, a web application login form, an FTP daemon.

sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd

Check that status output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, journalctl -u fail2ban will tell you why.

Set ignoreip in jail.local to include your office or home address. Banning yourself is a rite of passage but it is avoidable.

6. The boring items that matter more than the exciting ones

Nobody writes blog posts about these, and they cause more real outages than any of the above.

Provider account security

Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.

Time and timezone

sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"

UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.

Swap on small instances

# A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

Backups that leave the machine

Your provider’s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.

Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man’s switch service at the backup job so you find out when it stops running, rather than the day you need it.

Knowing the box is alive

At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.

What I would skip

Not everything in the long hardening guides earns its cost on a single web server.

• Large sysctl hardening blocks pasted from a gist. Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.
• AIDE, auditd and file-integrity monitoring on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.
• Rebuilding OpenSSH’s cipher list. Current defaults are good. Guides that hand you a Ciphers line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.
• Disabling ping. It breaks your own monitoring and hides nothing.

Troubleshooting

Locked out after an SSH change

Provider console, log in as root, and revert. If you cannot remember what you changed, ls -lt /etc/ssh/sshd_config.d/ shows the most recently modified file. Run sshd -t to see whether the config is even valid, since a syntax error stops sshd starting at all.

Key auth rejected, password prompt appears instead

Almost always permissions. The home directory must not be group-writable, ~/.ssh needs 700 and authorized_keys 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run ssh -v to see how far it gets, and check journalctl -u ssh on the server for the actual reason.

The setting I applied is not in effect

sudo sshd -T and search for the keyword. If the effective value differs from your file, something in sshd_config.d/ sorts earlier, or a Match block later in the config is overriding it for your connection.

A service is reachable that the firewall says is blocked

Docker, or another daemon writing netfilter rules directly. Check with sudo iptables -S or sudo nft list ruleset, and always verify from a machine outside the server rather than trusting ufw status.

Nothing obviously wrong, but who has been logging in?

last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"

Common mistakes

• Closing your only SSH session before testing the new configuration from a second one.
• Enabling the firewall before adding the SSH rule.
• Naming your sshd drop-in 99-, so a vendor file that sorts earlier wins.
• Trusting the file you edited instead of running sshd -T.
• Generating the SSH key on the server.
• Changing the SSH port and counting it as a security control.
• Disabling password auth for SSH while leaving a web control panel with a weak password exposed.
• No two-factor authentication on the hosting account that can reset root.
• Treating provider snapshots as backups.
• Installing fail2ban and never checking that it started.
• Publishing container ports without binding them to localhost, then assuming ufw covers it.
• Pasting a sysctl block you cannot explain line by line.

Best practices

• Verify every control after applying it, with a command that reads the running state.
• Keep a second session open and the provider console tested before making SSH changes.
• Key-only authentication, root login disabled, and an explicit AllowUsers list.
• Default-deny inbound firewall, with rules added before the firewall is enabled.
• Automatic security updates on, with a deliberate decision about automatic reboots.
• Two-factor authentication on the hosting account and the registrar.
• Backups off the machine, on separate credentials, and restore-tested at least once.
• UTC everywhere, with clock sync confirmed.
• Monitoring that pages you on disk usage before it pages you on downtime.
• Script the whole build, so the next server is identical and you are not repeating this from memory.

FAQ

What is the single most important step?

Key-only SSH with root login disabled, verified with sshd -T. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.

Should I change the SSH port?

For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.

Do I need fail2ban if password authentication is off?

Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.

Why did my sshd_config change get ignored?

Two likely causes. A file in /etc/ssh/sshd_config.d/ that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is Port or ListenAddress on a socket-activated system, where systemd owns the listener. sshd -T answers the first case and ss -tlnp answers the second.

Is ufw enough, or should I use nftables directly?

ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.

How do I harden a server I did not build?

Audit before you change anything. sshd -T, ufw status verbose, ss -tlnp for unexpected listeners, systemctl list-units --type=service --state=running, and the contents of every authorized_keys on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.

Should this be automated?

Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.

The one thing to remember

Editing a config file is not the same as changing a system’s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.

So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. sshd -T for SSH, ufw status verbose for the firewall, ss -tlnp for what is listening, systemctl status for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.

Want a second pair of eyes on your server?

Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:

• Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.
• Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.
• Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.
• Moving SSH off the public internet onto a private overlay network or bastion.
• Backup strategy that survives the hosting account being compromised, including a tested restore.
• Firewall and container networking, especially where published Docker ports are bypassing the firewall.

Send me the output of sshd -T, ufw status verbose and ss -tlnp, and I will tell you what stands out.

The post You Probably Didn’t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself appeared first on John Nessime.

docker compose up -d

A web application, database, cache, monitoring service, and reverse proxy can all be launched in seconds. Everything is isolated, repeatable, and relatively easy to move between environments.

Then, eventually, something breaks.

A container refuses to start. A port is already occupied. A database loses access to its storage directory. One service cannot resolve the hostname of another. A reverse proxy returns a 502 error even though the application appears to be running.

That is usually the point where Docker stops feeling like a convenient deployment tool and starts becoming a real infrastructure skill.

This guide is based on the way I approach Docker issues on real Linux servers. It is not a list of commands to copy blindly. It is a troubleshooting method for discovering what failed, understanding why it failed, and solving it without making the situation worse.

Docker Is Usually Not the Real Problem

When a container fails, Docker is often blamed first. In many cases, however, Docker is only exposing a problem somewhere else in the system.

• Incorrect Linux file permissions
• A missing environment variable
• An unavailable database
• A port conflict on the host
• An incorrect volume mount
• A DNS or network configuration problem
• A reverse proxy pointing to the wrong destination
• Insufficient memory or disk space
• An application-level configuration error

The container runtime may be working exactly as designed. The application inside the container may simply be unable to operate under the conditions it was given.

This distinction matters because repeatedly deleting and rebuilding containers rarely fixes the underlying issue. It may temporarily hide it, but the same failure usually returns.

My First Rule of Docker Troubleshooting: Do Not Delete Anything Yet

When a container suddenly exits, the first instinct is often to remove it, rebuild the image, or restart the entire stack. That destroys useful evidence.

Before changing anything, I start by collecting information.

docker ps -a

This shows running and stopped containers, their current states, exposed ports, names, and recent exit information.

docker logs container-name

The logs frequently reveal the immediate cause: a missing file, invalid password, failed database connection, permission error, or malformed configuration.

docker inspect container-name

Inspection provides the container’s network settings, environment variables, volume mounts, restart policy, health status, image information, and runtime configuration.

These three commands answer a large percentage of Docker troubleshooting questions before any corrective action is required.

Understanding Docker Container States

A container can exist in several states, and each state tells a different story.

Running

The main process inside the container is active. This does not automatically mean the application is healthy or ready to receive traffic.

Exited

The container’s main process stopped. It may have completed normally, crashed, or failed during startup. The exit code and logs provide the next clues.

Restarting

The container repeatedly starts and fails while Docker applies its restart policy. This commonly indicates an application crash, invalid configuration, inaccessible dependency, or permission problem.

Created

The container was created but its main process has not started successfully.

Dead

Docker could not properly stop or remove the container. This is less common and may indicate a lower-level runtime, storage, or host issue.

Knowing the state prevents random troubleshooting. A networking issue requires a different investigation from a process that exits because of a syntax error.

Start With the Container Logs

Logs are normally the fastest route to the cause of a Docker issue.

docker logs --tail 100 container-name

This displays the latest 100 lines without flooding the terminal with the container’s entire history.

docker logs -f container-name

The follow option streams new log entries in real time. It is useful while restarting a service or reproducing a problem.

docker logs --since 30m container-name

This limits the output to a recent period, which is especially useful for long-running containers.

When reading logs, I do not look only at the final line. The true cause often appears several lines earlier. The last error may simply be a consequence of the first failure.

Docker Networking: Where Many Problems Begin

Docker networking is one of the most common sources of confusion, especially when several containers need to communicate.

Imagine a stack containing:

• Grafana
• Prometheus
• A web application
• MySQL or PostgreSQL
• Redis
• Nginx or Apache

Each service may work independently while communication between them fails.

The Localhost Mistake

Inside a container, localhost refers to that container itself. It does not refer to the Docker host, and it does not refer to another container.

For example, if Grafana needs to connect to Prometheus, the data source address should usually use the Docker Compose service name:

http://prometheus:9090

Using the following from inside the Grafana container would normally point back to Grafana itself:

http://localhost:9090

This small misunderstanding causes a surprising number of connection failures.

Confirm That Containers Share a Network

docker network ls

docker network inspect network-name

The inspection output shows which containers are attached to a network and which IP addresses were assigned.

Containers on unrelated networks cannot automatically communicate by service name. They must share a Docker network or use another explicitly configured route.

Test Connectivity From Inside the Container

docker exec -it container-name sh

Depending on the image, Bash may be available instead:

docker exec -it container-name bash

From inside the container, test name resolution and connectivity using the utilities available in that image:

getent hosts database
curl http://application:8080
wget -qO- http://prometheus:9090/-/ready

A successful result confirms that Docker DNS and network routing are functioning. A failed result narrows the investigation considerably.

Host Ports and Container Ports Are Different

A Docker Compose port mapping such as the following is read from left to right:

ports:
  - "8080:80"

Port 8080 belongs to the Docker host. Port 80 belongs to the application inside the container.

Traffic sent to the host on port 8080 is forwarded to port 80 inside the container.

If another service already occupies host port 8080, Docker cannot bind to it.

ss -tulpn

To check a specific port:

ss -tulpn | grep :8080

A common mistake is changing the container’s internal port when only the host-side mapping needs to change.

For example, this avoids a conflict while leaving the application untouched:

ports:
  - "8081:80"

Published Ports Are Not Required for Container-to-Container Traffic

Containers connected to the same Docker network can usually communicate through their internal ports without publishing those ports to the host.

For example, Prometheus can communicate with an exporter on the shared Docker network without exposing that exporter publicly.

This reduces unnecessary exposure and produces a cleaner architecture. Publish only the ports that must be reached from the host, a reverse proxy, or an external client.

Docker Volumes: Persistent Storage Done Properly

Containers should be treated as disposable. Persistent data should not depend on the writable filesystem inside a container.

Without a correctly configured volume, deleting a database container may also delete the database stored inside it.

Named Volumes

services:
  database:
    image: mariadb:latest
    volumes:
      - database-data:/var/lib/mysql

volumes:
  database-data:

Docker manages the storage location while the application writes to its expected internal directory.

Bind Mounts

volumes:
  - ./config:/etc/application

A bind mount maps a specific host path into the container. It is useful for configuration files, development directories, and data that administrators need to access directly.

Bind mounts also introduce more responsibility. The host directory must exist with the correct permissions, ownership, security context, and contents.

Inspect Existing Volumes

docker volume ls

docker volume inspect volume-name

Before deleting an apparently unused volume, confirm which application created it and whether it contains irreplaceable data.

Permission Problems Are Quiet but Destructive

Permission errors can prevent databases from initializing, web applications from uploading files, monitoring tools from writing data, or services from reading configuration files.

The first checks are simple:

ls -lah
stat path-to-file
stat path-to-directory

The user inside the container may have a different numeric user ID from the owner of the host directory. The names do not matter as much as the numeric UID and GID.

docker exec container-name id

Compare that result with the ownership of the mounted host path.

Changing permissions to 777 may appear to solve the issue, but it is usually an unsafe shortcut rather than a proper solution. Correct ownership and the minimum required permissions are preferable.

SELinux Can Affect Docker Bind Mounts

On distributions such as AlmaLinux, Rocky Linux, CentOS Stream, Fedora, and Red Hat Enterprise Linux, standard Unix permissions may look correct while SELinux still blocks access.

Docker Compose bind mounts may require an SELinux relabeling option:

volumes:
  - ./data:/var/lib/application:Z

The uppercase Z assigns a private SELinux label for the container. A lowercase z is used when the content must be shared between multiple containers.

SELinux should not be disabled simply to make a container work. The better approach is to identify the denied access and apply the correct labeling or policy.

Environment Variables Can Break an Entire Stack

Modern containerized applications depend heavily on environment variables for database credentials, API keys, application URLs, encryption secrets, feature flags, and runtime settings.

One missing or incorrectly escaped value can stop a service from starting.

docker compose config

This command renders the resolved Docker Compose configuration. It helps reveal missing substitutions, merged settings, malformed values, and unexpected defaults.

To inspect the environment available inside a running container:

docker exec container-name env

Be careful when sharing this output because it may include passwords, tokens, and private service credentials.

Docker Compose Configuration Errors

YAML is readable, but it is sensitive to indentation and structure. A service can be valid YAML while still being configured incorrectly for Docker Compose.

Before deploying a change, validate the file:

docker compose config --quiet

If the command returns without an error, the configuration is structurally valid.

For a complete rendered version:

docker compose config

This is especially valuable when Compose files use environment substitutions, overrides, extension fields, or multiple configuration files.

Health Checks Reveal More Than Running Status

A container can be running while the application inside it is unusable.

A database may still be initializing. A web service may be listening but unable to connect to its database. An API may be running but returning errors for every request.

A health check lets Docker test actual application readiness.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

The application should expose a lightweight endpoint that confirms its critical dependencies are available.

To inspect the current status:

docker inspect --format='{{json .State.Health}}' container-name

A health check does not repair an unhealthy service by itself, but it provides a far more accurate operational signal than process status alone.

Restart Policies Can Hide Repeated Failures

Restart policies are useful in production, but they can create the impression that a service is stable when it is actually crashing continuously.

restart: unless-stopped

If the application exits every few seconds, Docker restarts it every few seconds. The container may appear in the process list, but the service may never become ready.

Check the restart count:

docker inspect --format='{{.RestartCount}}' container-name

A rapidly increasing number indicates that the restart policy is masking an unresolved failure.

Reverse Proxy Errors: 502, 504, SSL Problems, and Redirect Loops

Reverse proxies such as Nginx, Apache, Traefik, and Caddy often sit between users and containerized applications.

When the public website fails, the application container may still be perfectly healthy. The proxy may simply be unable to reach it.

Common Reverse Proxy Problems

• The upstream hostname is incorrect.
• The proxy uses the host port when it should use the container port.
• The proxy and application do not share a Docker network.
• The application listens only on 127.0.0.1 inside the container.
• The proxy forwards HTTP while the backend expects HTTPS.
• The application is unaware that the original request used HTTPS.
• The public hostname does not match the certificate.
• Both the proxy and application force redirects, creating a loop.

Test the application directly before blaming the proxy:

curl -I http://127.0.0.1:published-port

If the application responds directly but fails through the domain, the reverse proxy, firewall, DNS, or TLS configuration becomes the main area of investigation.

An Application Must Listen on the Correct Interface

An application inside a container may start successfully while listening only on 127.0.0.1. In that situation, Docker’s network interface may not be able to reach it.

Containerized web applications should usually listen on:

0.0.0.0

This allows the process to accept connections through the container’s network interfaces.

This setting is commonly controlled by application arguments or environment variables such as HOST, BIND_ADDRESS, or LISTEN_ADDRESS.

Resource Exhaustion Can Look Like a Docker Failure

Containers share the CPU, memory, storage, and network resources of the host. One service can consume enough resources to affect every other workload.

docker stats

This provides a live view of CPU, memory, network, and block I/O usage for running containers.

On the host, I also check:

free -h
df -h
df -i
uptime

Disk space is not the only storage limit. A filesystem can also run out of inodes, preventing new files from being created even when gigabytes remain available.

Kernel logs may reveal out-of-memory termination:

dmesg | grep -i -E "out of memory|killed process|oom"

If the kernel killed the process, the application logs may end suddenly without a useful explanation.

Docker Disk Usage Grows Quietly

Old images, stopped containers, build cache, unused volumes, and unbounded logs can gradually consume the host’s storage.

docker system df

For more detail:

docker system df -v

Cleanup commands should be used carefully:

docker image prune
docker container prune
docker builder prune

A broad command such as docker system prune can remove more than expected. Never add the volume-removal option unless the contents of unused volumes have been verified and backed up.

Container Logs Can Fill the Server

Docker’s default JSON logging driver can create large log files when applications produce continuous output.

Log rotation can be configured in Docker Compose:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

This limits the number and size of local log files for that container.

For production systems, centralized logging with tools such as Grafana Loki, Elasticsearch, OpenSearch, or a managed logging service makes incident investigation considerably easier.

Image Architecture and Compatibility Problems

An image built for one CPU architecture may not run correctly on another. This becomes relevant when moving between AMD64 servers, ARM-based cloud instances, Apple Silicon machines, and single-board computers.

Check the host architecture:

uname -m

Inspect the image:

docker image inspect image-name

Errors mentioning an invalid executable format frequently indicate an architecture mismatch.

Multi-platform builds can be produced using Docker Buildx:

docker buildx build --platform linux/amd64,linux/arm64 .

A New Image Tag Does Not Always Mean a Safe Upgrade

Using latest may be convenient, but it makes deployments less predictable. A future image update can introduce a breaking configuration change, database migration, removed feature, or incompatible dependency.

Production deployments are usually safer with an explicit version:

image: grafana/grafana:12.0.2

Before upgrading:

• Read the release notes.
• Confirm supported upgrade paths.
• Back up persistent data.
• Record the current working version.
• Test the change outside production where possible.
• Prepare a rollback procedure.

Containers make software easier to replace, but they do not make every application upgrade reversible.

Dependency Startup Order Is Not Readiness

Docker Compose can start services in a defined order, but starting a database container does not mean the database is immediately ready for connections.

An application may launch too early, fail to connect, and exit.

Health checks and dependency conditions can improve this:

services:
  database:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  application:
    image: example/application:1.0
    depends_on:
      database:
        condition: service_healthy

The application itself should still handle temporary dependency failures gracefully. Networks fluctuate, databases restart, and external services occasionally become unavailable.

Backups Matter More Than Recreating Containers

Application containers are normally easy to recreate. Persistent business data is not.

A reliable Docker backup strategy should cover:

• Database-native backups
• Named volumes and bind-mounted data
• Docker Compose files
• Environment files and secrets
• Reverse proxy configurations
• TLS and certificate configuration
• Application-specific settings
• A tested restoration procedure

A backup that has never been restored is only an assumption.

Database backups should normally use the database’s own tools rather than copying active database files blindly. Examples include pg_dump for PostgreSQL and mysqldump or mariadb-dump for MySQL-compatible systems.

Monitoring Changes Troubleshooting From Guesswork to Evidence

Without monitoring, administrators usually investigate after users report a failure. By then, the container may have restarted and the original conditions may no longer be visible.

Metrics help answer better questions:

• When did memory consumption begin increasing?
• Which container experienced the first error?
• Was the disk already nearly full?
• Did network latency rise before the application failed?
• How many times did the service restart?
• Did the host run out of memory?
• Was the reverse proxy still able to reach the backend?

Prometheus, Grafana, node-level exporters, container metrics, uptime probes, and centralized logs provide the historical context that individual Docker commands cannot.

A Practical Docker Troubleshooting Workflow

When a Docker service fails, I use a sequence rather than jumping between unrelated commands.

1. Identify the Exact Symptom

• Does the container fail to start?
• Does it start and then exit?
• Is it running but unreachable?
• Is the application responding with errors?
• Is the public domain failing while the local service works?
• Is the problem intermittent?

2. Check Container Status

docker ps -a

3. Read the Logs

docker logs --tail 200 container-name

4. Inspect Runtime Configuration

docker inspect container-name

5. Validate Docker Compose

docker compose config

6. Check Host Resources

docker stats
free -h
df -h
df -i

7. Test Internal Connectivity

Enter the relevant container and test DNS resolution, ports, and HTTP endpoints from the same network context as the application.

8. Check Volumes and Permissions

Confirm that mounts point to the expected locations and that the container user can read or write them as required.

9. Isolate the Reverse Proxy

Test the backend directly. If the backend works, investigate the proxy, DNS, TLS, firewall, and forwarded headers.

10. Change One Thing at a Time

Multiple simultaneous changes make it difficult to identify the actual solution. Apply one correction, test it, and record the result.

Commands I Frequently Use During Docker Investigations

# Show running containers
docker ps

# Show all containers
docker ps -a

# Read recent logs
docker logs --tail 100 container-name

# Follow logs
docker logs -f container-name

# Inspect a container
docker inspect container-name

# Enter a container
docker exec -it container-name sh

# View live resource use
docker stats

# List networks
docker network ls

# Inspect a network
docker network inspect network-name

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect volume-name

# Validate Compose configuration
docker compose config

# View Compose service status
docker compose ps

# Restart one service
docker compose restart service-name

# Recreate one service
docker compose up -d --force-recreate service-name

# View Docker disk usage
docker system df

# View host listening ports
ss -tulpn

# Check disk usage
df -h

# Check inode usage
df -i

# Check memory
free -h

Docker Troubleshooting Mistakes to Avoid

Deleting Containers Before Reading Their Logs

This removes evidence and often leaves the original problem unresolved.

Using Docker System Prune Without Reviewing the Impact

Cleanup commands can remove useful images, build cache, stopped containers, and potentially important storage.

Using Latest Everywhere

Uncontrolled image updates reduce reproducibility and can introduce breaking changes unexpectedly.

Opening Every Port Publicly

Internal services should remain internal unless external access is genuinely required.

Setting Permissions to 777

This weakens security and avoids fixing the actual ownership or access-control issue.

Restarting the Entire Server Too Early

A restart may temporarily restore service while destroying evidence about the original failure.

Changing Several Variables at Once

Even when the issue disappears, it becomes impossible to know which change solved it.

Docker Compose Is Also Infrastructure Documentation

A well-organized Compose file documents the structure of an application environment.

• Which services exist
• Which image versions are deployed
• Which networks connect them
• Which ports are externally accessible
• Where persistent data is stored
• Which environment variables are required
• Which services depend on others
• Which health checks are available
• How containers restart after failure

Six months after a deployment, this information is often more useful than a separate document that was never updated.

Readable service names, comments for non-obvious decisions, pinned image versions, organized environment files, and clear volume names make future troubleshooting much faster.

What Docker Failures Actually Teach

Some of the most useful infrastructure lessons come from deployments that did not work as expected.

A failed container may teach you how Linux permissions interact with numeric user IDs. A 502 error may reveal the difference between host ports and container ports. A disappearing database may demonstrate why persistent volumes matter. An unreachable service may expose a misunderstanding about Docker DNS or network boundaries.

These incidents are frustrating, but they build practical knowledge that clean tutorial environments rarely provide.

Production systems do not normally fail in textbook ways. Several small issues may combine: a nearly full disk, oversized logs, an automatic image upgrade, and an aggressive restart policy can create one confusing incident.

Final Thoughts

Docker makes application deployment more portable, repeatable, and manageable, but it does not remove the need to understand the systems underneath it.

Effective Docker troubleshooting requires knowledge of Linux, networking, storage, process management, application configuration, security, and monitoring.

The most important habit is simple: follow the evidence.

Check the container state. Read the logs. Inspect the configuration. Test the network from the correct location. Verify volumes and permissions. Check the host’s resources. Separate the application from the proxy. Change one thing at a time.

The next time a Docker container fails, resist the urge to delete everything and begin again. The answer is usually already present in the logs, runtime configuration, network design, storage mapping, or host environment.

Docker troubleshooting becomes much less intimidating once every failure is treated as an investigation rather than an emergency.

Frequently Asked Questions About Docker Issues

Why does my Docker container exit immediately?

A Docker container exits when its main process stops. Common causes include invalid configuration, missing environment variables, permission errors, failed dependency connections, and incorrect startup commands. Use docker logs and docker inspect to identify the cause.

Why is my Docker container running but not accessible?

The application may be listening on the wrong interface, the port may not be published correctly, a firewall may be blocking traffic, or the reverse proxy may point to the wrong hostname or port. Test the application both inside the container and directly through the host port.

Why can one Docker container not connect to another?

The containers may not share a Docker network, the wrong service name may be used, or the destination service may not be listening on the expected port. Inside Docker, use the Compose service name rather than localhost.

How do I find which process is using a Docker port?

Run ss -tulpn on the Docker host and filter for the relevant port. If the host port is already occupied, change the host side of the Docker port mapping or stop the conflicting service.

Can restarting Docker solve container problems?

Restarting Docker may temporarily restore a service, but it can also hide the original cause. Logs, container state, disk space, memory, network connectivity, and volume permissions should be checked before restarting the Docker daemon or server.

How can I prevent Docker logs from filling the disk?

Configure log rotation using the Docker logging options, such as max-size and max-file. Production environments may also benefit from centralized logging with Grafana Loki, OpenSearch, Elasticsearch, or a managed logging platform.

Should I use the latest Docker image tag?

The latest tag is convenient but unpredictable. Pinning a tested image version improves repeatability and makes upgrades and rollbacks easier to control.

Are Docker volumes automatically backed up?

No. Docker volumes provide persistent storage, but Docker does not automatically create external backups. Important volumes and databases need a separate, tested backup and restoration process.

The post Docker Issues & Troubleshooting: Real-World Solutions appeared first on John Nessime.