That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.

Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.

This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.

What Docker Compose genuinely gets right

Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.

• The whole system fits in one file you can read. Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.
• Dev and prod can share a base file. Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.
• No control plane to operate. A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.
• Recovery is comprehensible. When something breaks at 2am, docker compose ps and docker compose logs tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.

If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.

Failure family one: the deploy gap

This is the one that surprises people who came from a platform that did rolling updates for them.

docker compose up -d is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.

The deploy.update_config block with order: start-first exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.

What to do instead

Three options, in increasing order of effort:

1. Accept the gap and shrink it. Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.
2. Put a reverse proxy in front and run two slots. Traefik, Caddy or plain Nginx in a container, with app-blue and app-green services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.
3. Drain at the edge. If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.

Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:

# Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.

That last line matters more than it looks. A service with no health check is invisible to --wait, so a deploy script can report success while your API is still crash-looping.

Failure family two: the disk fills up and nothing tells you

Docker’s default logging driver is json-file, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local driver is the recommended alternative because it rotates out of the box and uses a more compact format.

So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.

Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}

Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.

You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:

x-logging: &default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    <<: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    <<: *default-logging

Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.

# Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v

Failure family three: state you did not mean to keep

Two related traps here.

The first is anonymous volumes. If an image declares a VOLUME in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.

The second is docker compose down. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.

The practical rule: on a production host, restarting services is docker compose up -d or docker compose restart. down belongs in a maintenance procedure with a fresh backup, not in a daily habit.

Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.

Failure family four: ports that bypass your firewall

This is the one that turns into a security incident rather than an outage.

When you write ports: - "5432:5432", Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status output that looks perfectly reassuring.

The fix is to stop publishing what does not need publishing:

services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]

Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.

While you are in daemon config: set default-address-pools as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.

Failure family five: secrets in environment variables

Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.

The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under /run/secrets/:

services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.

One more thing that trips people up: .env and env_file are different mechanisms. The .env file in the project directory feeds variable interpolation inside the Compose file. env_file passes variables into the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.

Failure family six: health checks that report but never act

A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.

So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.

Health checks are still worth defining, because they gate startup ordering and they gate --wait. Just be clear about what they do not do.

services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s

Note restart: unless-stopped rather than always. The difference shows up after a host reboot: always will start a container you had deliberately stopped, unless-stopped respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.

To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.

Troubleshooting a Compose stack that is misbehaving

The commands I reach for, roughly in order:

# The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' <container>

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"

Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.

Common mistakes

• Deploying :latest. You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.
• Keeping the version: key at the top of the file. Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.
• No resource limits anywhere. One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.
• Running docker compose up from an SSH session and walking away. Without -d the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.
• The Compose file lives only on the server. If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.
• Pulling from Docker Hub anonymously in a deploy script. Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.
• Automatic image updates on production. Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.

Best practices for Docker Compose in production

• One base file plus a production override. Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then docker compose -f compose.yaml -f compose.prod.yaml up -d is your deploy, and the diff between environments is a file you can read.
• Give every service a health check. It gates startup order, it makes --wait meaningful, and it gives you a status worth alerting on.
• Set memory and CPU limits on everything. Watch docker stats under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.
• Segment your networks. An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.
• Back up volumes, and restore one. A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.
• Make the deploy a script, not a memory. Pull, up with --wait, verify, prune. Six lines in the repo beats six commands somebody half-remembers.
• Run docker compose config in CI. It catches a malformed override before it reaches the server, and it costs nothing.

When to stop using Compose

Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.

The signals I treat as a real trigger, rather than as an excuse to rewrite everything:

• You need more than one machine for availability, not just for capacity.
• You need automatic rescheduling when a host dies, without somebody logging in.
• You are scaling components independently and often enough that doing it by hand is a real cost.
• Several teams deploy to the same infrastructure and need isolation from each other.

If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.

Frequently asked questions

Is Docker Compose production ready?

For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.

How do I get zero-downtime deployments with Docker Compose?

Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.

Does docker compose down delete my database?

Not by itself. Plain down removes containers and the project’s networks and leaves named volumes in place. Adding -v removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.

Why does my container keep restarting with exit code 137?

137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats for growth over time and dmesg for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.

Should I still write version: at the top of my Compose file?

No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.

Can I use Docker secrets without Swarm?

Yes, using file-based secrets. Declare a top-level secrets block with a file: source, grant it to a service, and the content appears at /run/secrets/<name> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect.

How many services is too many for one Compose file?

There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.

The one thing worth remembering

Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.

Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.

Need a second pair of eyes on your Compose setup?

I work with teams running containerised workloads on their own servers. Things I am usually brought in for:

• Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first
• Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI
• Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does
• Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall
• Volume backup and restore that has been tested by restoring, not just by running
• An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down

If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.

The post Docker Compose in Production: What Works and What Quietly Burns You appeared first on John Nessime.

Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was Delete, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.

That is the sharp edge of Kubernetes persistent volumes, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.

This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.

Three objects, one disk

The mental model that prevents most of this:

• StorageClass is the recipe. It says which driver provisions volumes and what the defaults are.
• PersistentVolume is the actual piece of storage in the cluster. Usually created for you rather than by you.
• PersistentVolumeClaim is the request a pod makes, and the thing that binds one-to-one with a PV.

The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:

# Read the RECLAIMPOLICY column. On managed clusters it usually
# says Delete, and nothing anywhere warns you about that.
kubectl get storageclass

For anything holding real data, define your own class with Retain and use it deliberately:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain              # PV outlives the PVC
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3

The trade-off is real and worth stating: Retain means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.

Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:

kubectl patch pv "$PV_NAME" 
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

ReadWriteOnce does not mean one pod

It means one node. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.

For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:

spec:
  accessModes:
    - ReadWriteOncePod        # enforced by the scheduler

The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.

spec:
  strategy:
    type: Recreate            # old pod goes first, then the new one

That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.

Volumes pin pods to a zone

Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in Pending after a node replacement or a scale-up that happened to land elsewhere.

kubectl describe pod tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.

The preventive setting is volumeBindingMode: WaitForFirstConsumer, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.

StatefulSet or Deployment

The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same volume, which is only correct when you have one replica.

So: one instance with a disk can be a Deployment with Recreate. Anything that scales and keeps per-replica state should be a StatefulSet.

StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to Retain:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain       # deleting the StatefulSet keeps the data
    whenScaled: Delete        # scaling down cleans up the extra replicas

That pairing is the sensible default for most workloads. Be careful with whenScaled: Delete on anything where you scale down temporarily and expect the data back when you scale up again.

Two one-way doors

Volumes grow, they do not shrink. Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.

A volume snapshot is not a database backup. Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.

Troubleshooting

Pod stuck in Pending. Read the events at the bottom of kubectl describe pod. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.

PVC stuck in Pending. kubectl describe pvc names the provisioner error. With WaitForFirstConsumer this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.

Rollout hangs forever on a stateful app. ReadWriteOnce plus rolling update. Switch to Recreate.

PVC will not delete. A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.

Volume is full but the PVC says it has room. You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.

Common mistakes

• Running production data on the default StorageClass without checking its reclaim policy.
• Assuming ReadWriteOnce prevents two pods from writing.
• A ReadWriteOnce volume on a Deployment with rolling updates.
• Scaling a Deployment with a PVC past one replica.
• Leaving volumeBindingMode: Immediate, then debugging Pending pods for an hour.
• Treating volume snapshots as database backups.
• Over-sizing a PVC on the assumption you can shrink it later.
• Removing a stuck finalizer instead of finding what is holding the volume.
• Deleting a namespace without checking what claims live in it.

Best practices

• Your own StorageClass with Retain for anything you would miss, and the default for everything else.
• WaitForFirstConsumer on every class you define.
• allowVolumeExpansion: true from the start, because adding it later does not help existing volumes.
• ReadWriteOncePod where exactly one writer is a correctness requirement.
• StatefulSets for per-replica state, single-replica Deployments with Recreate for everything else.
• An explicit persistentVolumeClaimRetentionPolicy rather than relying on the default.
• Application-level backups for databases, tested by restoring one.
• Alerts on volume capacity, because a full disk is the most predictable outage there is.

FAQ

Does deleting a PVC delete my data?

If the reclaim policy is Delete, yes, and that is the default nearly everywhere. Run kubectl get storageclass and read the RECLAIMPOLICY column before you assume otherwise.

Can two pods share a ReadWriteOnce volume?

Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.

Why is my pod Pending with a bound PVC?

Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in kubectl describe pod say so. WaitForFirstConsumer prevents it happening again.

Should I run databases in Kubernetes at all?

You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.

Can I shrink a PersistentVolumeClaim?

No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.

The one thing to remember

Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.

Go and run kubectl get storageclass on your production cluster now. If the reclaim policy on the class holding your database says Delete, you have found something worth fixing before lunch.

Need a hand with stateful workloads?

Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:

• Auditing a cluster’s storage classes, reclaim policies and access modes, and telling you what a stray kubectl delete would actually destroy.
• Moving stateful applications onto StatefulSets with sane retention, expansion and scheduling behaviour.
• Backup and restore design for data in Kubernetes, including an actual restore test rather than a documented intention.
• Diagnosing Pending pods, stuck rollouts and volumes that will not detach.
• CSI driver setup and storage class design on EKS and self-managed clusters.
• Capacity alerting so a full volume is a notification instead of an incident.

Send me the output of kubectl get storageclass and kubectl get pv, and I will tell you what stands out.

The post It’s Only a Claim: Kubernetes Persistent Volumes and Stateful Workloads appeared first on John Nessime.

docker compose up -d

A web application, database, cache, monitoring service, and reverse proxy can all be launched in seconds. Everything is isolated, repeatable, and relatively easy to move between environments.

Then, eventually, something breaks.

A container refuses to start. A port is already occupied. A database loses access to its storage directory. One service cannot resolve the hostname of another. A reverse proxy returns a 502 error even though the application appears to be running.

That is usually the point where Docker stops feeling like a convenient deployment tool and starts becoming a real infrastructure skill.

This guide is based on the way I approach Docker issues on real Linux servers. It is not a list of commands to copy blindly. It is a troubleshooting method for discovering what failed, understanding why it failed, and solving it without making the situation worse.

Docker Is Usually Not the Real Problem

When a container fails, Docker is often blamed first. In many cases, however, Docker is only exposing a problem somewhere else in the system.

• Incorrect Linux file permissions
• A missing environment variable
• An unavailable database
• A port conflict on the host
• An incorrect volume mount
• A DNS or network configuration problem
• A reverse proxy pointing to the wrong destination
• Insufficient memory or disk space
• An application-level configuration error

The container runtime may be working exactly as designed. The application inside the container may simply be unable to operate under the conditions it was given.

This distinction matters because repeatedly deleting and rebuilding containers rarely fixes the underlying issue. It may temporarily hide it, but the same failure usually returns.

My First Rule of Docker Troubleshooting: Do Not Delete Anything Yet

When a container suddenly exits, the first instinct is often to remove it, rebuild the image, or restart the entire stack. That destroys useful evidence.

Before changing anything, I start by collecting information.

docker ps -a

This shows running and stopped containers, their current states, exposed ports, names, and recent exit information.

docker logs container-name

The logs frequently reveal the immediate cause: a missing file, invalid password, failed database connection, permission error, or malformed configuration.

docker inspect container-name

Inspection provides the container’s network settings, environment variables, volume mounts, restart policy, health status, image information, and runtime configuration.

These three commands answer a large percentage of Docker troubleshooting questions before any corrective action is required.

Understanding Docker Container States

A container can exist in several states, and each state tells a different story.

Running

The main process inside the container is active. This does not automatically mean the application is healthy or ready to receive traffic.

Exited

The container’s main process stopped. It may have completed normally, crashed, or failed during startup. The exit code and logs provide the next clues.

Restarting

The container repeatedly starts and fails while Docker applies its restart policy. This commonly indicates an application crash, invalid configuration, inaccessible dependency, or permission problem.

Created

The container was created but its main process has not started successfully.

Dead

Docker could not properly stop or remove the container. This is less common and may indicate a lower-level runtime, storage, or host issue.

Knowing the state prevents random troubleshooting. A networking issue requires a different investigation from a process that exits because of a syntax error.

Start With the Container Logs

Logs are normally the fastest route to the cause of a Docker issue.

docker logs --tail 100 container-name

This displays the latest 100 lines without flooding the terminal with the container’s entire history.

docker logs -f container-name

The follow option streams new log entries in real time. It is useful while restarting a service or reproducing a problem.

docker logs --since 30m container-name

This limits the output to a recent period, which is especially useful for long-running containers.

When reading logs, I do not look only at the final line. The true cause often appears several lines earlier. The last error may simply be a consequence of the first failure.

Docker Networking: Where Many Problems Begin

Docker networking is one of the most common sources of confusion, especially when several containers need to communicate.

Imagine a stack containing:

• Grafana
• Prometheus
• A web application
• MySQL or PostgreSQL
• Redis
• Nginx or Apache

Each service may work independently while communication between them fails.

The Localhost Mistake

Inside a container, localhost refers to that container itself. It does not refer to the Docker host, and it does not refer to another container.

For example, if Grafana needs to connect to Prometheus, the data source address should usually use the Docker Compose service name:

http://prometheus:9090

Using the following from inside the Grafana container would normally point back to Grafana itself:

http://localhost:9090

This small misunderstanding causes a surprising number of connection failures.

Confirm That Containers Share a Network

docker network ls

docker network inspect network-name

The inspection output shows which containers are attached to a network and which IP addresses were assigned.

Containers on unrelated networks cannot automatically communicate by service name. They must share a Docker network or use another explicitly configured route.

Test Connectivity From Inside the Container

docker exec -it container-name sh

Depending on the image, Bash may be available instead:

docker exec -it container-name bash

From inside the container, test name resolution and connectivity using the utilities available in that image:

getent hosts database
curl http://application:8080
wget -qO- http://prometheus:9090/-/ready

A successful result confirms that Docker DNS and network routing are functioning. A failed result narrows the investigation considerably.

Host Ports and Container Ports Are Different

A Docker Compose port mapping such as the following is read from left to right:

ports:
  - "8080:80"

Port 8080 belongs to the Docker host. Port 80 belongs to the application inside the container.

Traffic sent to the host on port 8080 is forwarded to port 80 inside the container.

If another service already occupies host port 8080, Docker cannot bind to it.

ss -tulpn

To check a specific port:

ss -tulpn | grep :8080

A common mistake is changing the container’s internal port when only the host-side mapping needs to change.

For example, this avoids a conflict while leaving the application untouched:

ports:
  - "8081:80"

Published Ports Are Not Required for Container-to-Container Traffic

Containers connected to the same Docker network can usually communicate through their internal ports without publishing those ports to the host.

For example, Prometheus can communicate with an exporter on the shared Docker network without exposing that exporter publicly.

This reduces unnecessary exposure and produces a cleaner architecture. Publish only the ports that must be reached from the host, a reverse proxy, or an external client.

Docker Volumes: Persistent Storage Done Properly

Containers should be treated as disposable. Persistent data should not depend on the writable filesystem inside a container.

Without a correctly configured volume, deleting a database container may also delete the database stored inside it.

Named Volumes

services:
  database:
    image: mariadb:latest
    volumes:
      - database-data:/var/lib/mysql

volumes:
  database-data:

Docker manages the storage location while the application writes to its expected internal directory.

Bind Mounts

volumes:
  - ./config:/etc/application

A bind mount maps a specific host path into the container. It is useful for configuration files, development directories, and data that administrators need to access directly.

Bind mounts also introduce more responsibility. The host directory must exist with the correct permissions, ownership, security context, and contents.

Inspect Existing Volumes

docker volume ls

docker volume inspect volume-name

Before deleting an apparently unused volume, confirm which application created it and whether it contains irreplaceable data.

Permission Problems Are Quiet but Destructive

Permission errors can prevent databases from initializing, web applications from uploading files, monitoring tools from writing data, or services from reading configuration files.

The first checks are simple:

ls -lah
stat path-to-file
stat path-to-directory

The user inside the container may have a different numeric user ID from the owner of the host directory. The names do not matter as much as the numeric UID and GID.

docker exec container-name id

Compare that result with the ownership of the mounted host path.

Changing permissions to 777 may appear to solve the issue, but it is usually an unsafe shortcut rather than a proper solution. Correct ownership and the minimum required permissions are preferable.

SELinux Can Affect Docker Bind Mounts

On distributions such as AlmaLinux, Rocky Linux, CentOS Stream, Fedora, and Red Hat Enterprise Linux, standard Unix permissions may look correct while SELinux still blocks access.

Docker Compose bind mounts may require an SELinux relabeling option:

volumes:
  - ./data:/var/lib/application:Z

The uppercase Z assigns a private SELinux label for the container. A lowercase z is used when the content must be shared between multiple containers.

SELinux should not be disabled simply to make a container work. The better approach is to identify the denied access and apply the correct labeling or policy.

Environment Variables Can Break an Entire Stack

Modern containerized applications depend heavily on environment variables for database credentials, API keys, application URLs, encryption secrets, feature flags, and runtime settings.

One missing or incorrectly escaped value can stop a service from starting.

docker compose config

This command renders the resolved Docker Compose configuration. It helps reveal missing substitutions, merged settings, malformed values, and unexpected defaults.

To inspect the environment available inside a running container:

docker exec container-name env

Be careful when sharing this output because it may include passwords, tokens, and private service credentials.

Docker Compose Configuration Errors

YAML is readable, but it is sensitive to indentation and structure. A service can be valid YAML while still being configured incorrectly for Docker Compose.

Before deploying a change, validate the file:

docker compose config --quiet

If the command returns without an error, the configuration is structurally valid.

For a complete rendered version:

docker compose config

This is especially valuable when Compose files use environment substitutions, overrides, extension fields, or multiple configuration files.

Health Checks Reveal More Than Running Status

A container can be running while the application inside it is unusable.

A database may still be initializing. A web service may be listening but unable to connect to its database. An API may be running but returning errors for every request.

A health check lets Docker test actual application readiness.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

The application should expose a lightweight endpoint that confirms its critical dependencies are available.

To inspect the current status:

docker inspect --format='{{json .State.Health}}' container-name

A health check does not repair an unhealthy service by itself, but it provides a far more accurate operational signal than process status alone.

Restart Policies Can Hide Repeated Failures

Restart policies are useful in production, but they can create the impression that a service is stable when it is actually crashing continuously.

restart: unless-stopped

If the application exits every few seconds, Docker restarts it every few seconds. The container may appear in the process list, but the service may never become ready.

Check the restart count:

docker inspect --format='{{.RestartCount}}' container-name

A rapidly increasing number indicates that the restart policy is masking an unresolved failure.

Reverse Proxy Errors: 502, 504, SSL Problems, and Redirect Loops

Reverse proxies such as Nginx, Apache, Traefik, and Caddy often sit between users and containerized applications.

When the public website fails, the application container may still be perfectly healthy. The proxy may simply be unable to reach it.

Common Reverse Proxy Problems

• The upstream hostname is incorrect.
• The proxy uses the host port when it should use the container port.
• The proxy and application do not share a Docker network.
• The application listens only on 127.0.0.1 inside the container.
• The proxy forwards HTTP while the backend expects HTTPS.
• The application is unaware that the original request used HTTPS.
• The public hostname does not match the certificate.
• Both the proxy and application force redirects, creating a loop.

Test the application directly before blaming the proxy:

curl -I http://127.0.0.1:published-port

If the application responds directly but fails through the domain, the reverse proxy, firewall, DNS, or TLS configuration becomes the main area of investigation.

An Application Must Listen on the Correct Interface

An application inside a container may start successfully while listening only on 127.0.0.1. In that situation, Docker’s network interface may not be able to reach it.

Containerized web applications should usually listen on:

0.0.0.0

This allows the process to accept connections through the container’s network interfaces.

This setting is commonly controlled by application arguments or environment variables such as HOST, BIND_ADDRESS, or LISTEN_ADDRESS.

Resource Exhaustion Can Look Like a Docker Failure

Containers share the CPU, memory, storage, and network resources of the host. One service can consume enough resources to affect every other workload.

docker stats

This provides a live view of CPU, memory, network, and block I/O usage for running containers.

On the host, I also check:

free -h
df -h
df -i
uptime

Disk space is not the only storage limit. A filesystem can also run out of inodes, preventing new files from being created even when gigabytes remain available.

Kernel logs may reveal out-of-memory termination:

dmesg | grep -i -E "out of memory|killed process|oom"

If the kernel killed the process, the application logs may end suddenly without a useful explanation.

Docker Disk Usage Grows Quietly

Old images, stopped containers, build cache, unused volumes, and unbounded logs can gradually consume the host’s storage.

docker system df

For more detail:

docker system df -v

Cleanup commands should be used carefully:

docker image prune
docker container prune
docker builder prune

A broad command such as docker system prune can remove more than expected. Never add the volume-removal option unless the contents of unused volumes have been verified and backed up.

Container Logs Can Fill the Server

Docker’s default JSON logging driver can create large log files when applications produce continuous output.

Log rotation can be configured in Docker Compose:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

This limits the number and size of local log files for that container.

For production systems, centralized logging with tools such as Grafana Loki, Elasticsearch, OpenSearch, or a managed logging service makes incident investigation considerably easier.

Image Architecture and Compatibility Problems

An image built for one CPU architecture may not run correctly on another. This becomes relevant when moving between AMD64 servers, ARM-based cloud instances, Apple Silicon machines, and single-board computers.

Check the host architecture:

uname -m

Inspect the image:

docker image inspect image-name

Errors mentioning an invalid executable format frequently indicate an architecture mismatch.

Multi-platform builds can be produced using Docker Buildx:

docker buildx build --platform linux/amd64,linux/arm64 .

A New Image Tag Does Not Always Mean a Safe Upgrade

Using latest may be convenient, but it makes deployments less predictable. A future image update can introduce a breaking configuration change, database migration, removed feature, or incompatible dependency.

Production deployments are usually safer with an explicit version:

image: grafana/grafana:12.0.2

Before upgrading:

• Read the release notes.
• Confirm supported upgrade paths.
• Back up persistent data.
• Record the current working version.
• Test the change outside production where possible.
• Prepare a rollback procedure.

Containers make software easier to replace, but they do not make every application upgrade reversible.

Dependency Startup Order Is Not Readiness

Docker Compose can start services in a defined order, but starting a database container does not mean the database is immediately ready for connections.

An application may launch too early, fail to connect, and exit.

Health checks and dependency conditions can improve this:

services:
  database:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  application:
    image: example/application:1.0
    depends_on:
      database:
        condition: service_healthy

The application itself should still handle temporary dependency failures gracefully. Networks fluctuate, databases restart, and external services occasionally become unavailable.

Backups Matter More Than Recreating Containers

Application containers are normally easy to recreate. Persistent business data is not.

A reliable Docker backup strategy should cover:

• Database-native backups
• Named volumes and bind-mounted data
• Docker Compose files
• Environment files and secrets
• Reverse proxy configurations
• TLS and certificate configuration
• Application-specific settings
• A tested restoration procedure

A backup that has never been restored is only an assumption.

Database backups should normally use the database’s own tools rather than copying active database files blindly. Examples include pg_dump for PostgreSQL and mysqldump or mariadb-dump for MySQL-compatible systems.

Monitoring Changes Troubleshooting From Guesswork to Evidence

Without monitoring, administrators usually investigate after users report a failure. By then, the container may have restarted and the original conditions may no longer be visible.

Metrics help answer better questions:

• When did memory consumption begin increasing?
• Which container experienced the first error?
• Was the disk already nearly full?
• Did network latency rise before the application failed?
• How many times did the service restart?
• Did the host run out of memory?
• Was the reverse proxy still able to reach the backend?

Prometheus, Grafana, node-level exporters, container metrics, uptime probes, and centralized logs provide the historical context that individual Docker commands cannot.

A Practical Docker Troubleshooting Workflow

When a Docker service fails, I use a sequence rather than jumping between unrelated commands.

1. Identify the Exact Symptom

• Does the container fail to start?
• Does it start and then exit?
• Is it running but unreachable?
• Is the application responding with errors?
• Is the public domain failing while the local service works?
• Is the problem intermittent?

2. Check Container Status

docker ps -a

3. Read the Logs

docker logs --tail 200 container-name

4. Inspect Runtime Configuration

docker inspect container-name

5. Validate Docker Compose

docker compose config

6. Check Host Resources

docker stats
free -h
df -h
df -i

7. Test Internal Connectivity

Enter the relevant container and test DNS resolution, ports, and HTTP endpoints from the same network context as the application.

8. Check Volumes and Permissions

Confirm that mounts point to the expected locations and that the container user can read or write them as required.

9. Isolate the Reverse Proxy

Test the backend directly. If the backend works, investigate the proxy, DNS, TLS, firewall, and forwarded headers.

10. Change One Thing at a Time

Multiple simultaneous changes make it difficult to identify the actual solution. Apply one correction, test it, and record the result.

Commands I Frequently Use During Docker Investigations

# Show running containers
docker ps

# Show all containers
docker ps -a

# Read recent logs
docker logs --tail 100 container-name

# Follow logs
docker logs -f container-name

# Inspect a container
docker inspect container-name

# Enter a container
docker exec -it container-name sh

# View live resource use
docker stats

# List networks
docker network ls

# Inspect a network
docker network inspect network-name

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect volume-name

# Validate Compose configuration
docker compose config

# View Compose service status
docker compose ps

# Restart one service
docker compose restart service-name

# Recreate one service
docker compose up -d --force-recreate service-name

# View Docker disk usage
docker system df

# View host listening ports
ss -tulpn

# Check disk usage
df -h

# Check inode usage
df -i

# Check memory
free -h

Docker Troubleshooting Mistakes to Avoid

Deleting Containers Before Reading Their Logs

This removes evidence and often leaves the original problem unresolved.

Using Docker System Prune Without Reviewing the Impact

Cleanup commands can remove useful images, build cache, stopped containers, and potentially important storage.

Using Latest Everywhere

Uncontrolled image updates reduce reproducibility and can introduce breaking changes unexpectedly.

Opening Every Port Publicly

Internal services should remain internal unless external access is genuinely required.

Setting Permissions to 777

This weakens security and avoids fixing the actual ownership or access-control issue.

Restarting the Entire Server Too Early

A restart may temporarily restore service while destroying evidence about the original failure.

Changing Several Variables at Once

Even when the issue disappears, it becomes impossible to know which change solved it.

Docker Compose Is Also Infrastructure Documentation

A well-organized Compose file documents the structure of an application environment.

• Which services exist
• Which image versions are deployed
• Which networks connect them
• Which ports are externally accessible
• Where persistent data is stored
• Which environment variables are required
• Which services depend on others
• Which health checks are available
• How containers restart after failure

Six months after a deployment, this information is often more useful than a separate document that was never updated.

Readable service names, comments for non-obvious decisions, pinned image versions, organized environment files, and clear volume names make future troubleshooting much faster.

What Docker Failures Actually Teach

Some of the most useful infrastructure lessons come from deployments that did not work as expected.

A failed container may teach you how Linux permissions interact with numeric user IDs. A 502 error may reveal the difference between host ports and container ports. A disappearing database may demonstrate why persistent volumes matter. An unreachable service may expose a misunderstanding about Docker DNS or network boundaries.

These incidents are frustrating, but they build practical knowledge that clean tutorial environments rarely provide.

Production systems do not normally fail in textbook ways. Several small issues may combine: a nearly full disk, oversized logs, an automatic image upgrade, and an aggressive restart policy can create one confusing incident.

Final Thoughts

Docker makes application deployment more portable, repeatable, and manageable, but it does not remove the need to understand the systems underneath it.

Effective Docker troubleshooting requires knowledge of Linux, networking, storage, process management, application configuration, security, and monitoring.

The most important habit is simple: follow the evidence.

Check the container state. Read the logs. Inspect the configuration. Test the network from the correct location. Verify volumes and permissions. Check the host’s resources. Separate the application from the proxy. Change one thing at a time.

The next time a Docker container fails, resist the urge to delete everything and begin again. The answer is usually already present in the logs, runtime configuration, network design, storage mapping, or host environment.

Docker troubleshooting becomes much less intimidating once every failure is treated as an investigation rather than an emergency.

Frequently Asked Questions About Docker Issues

Why does my Docker container exit immediately?

A Docker container exits when its main process stops. Common causes include invalid configuration, missing environment variables, permission errors, failed dependency connections, and incorrect startup commands. Use docker logs and docker inspect to identify the cause.

Why is my Docker container running but not accessible?

The application may be listening on the wrong interface, the port may not be published correctly, a firewall may be blocking traffic, or the reverse proxy may point to the wrong hostname or port. Test the application both inside the container and directly through the host port.

Why can one Docker container not connect to another?

The containers may not share a Docker network, the wrong service name may be used, or the destination service may not be listening on the expected port. Inside Docker, use the Compose service name rather than localhost.

How do I find which process is using a Docker port?

Run ss -tulpn on the Docker host and filter for the relevant port. If the host port is already occupied, change the host side of the Docker port mapping or stop the conflicting service.

Can restarting Docker solve container problems?

Restarting Docker may temporarily restore a service, but it can also hide the original cause. Logs, container state, disk space, memory, network connectivity, and volume permissions should be checked before restarting the Docker daemon or server.

How can I prevent Docker logs from filling the disk?

Configure log rotation using the Docker logging options, such as max-size and max-file. Production environments may also benefit from centralized logging with Grafana Loki, OpenSearch, Elasticsearch, or a managed logging platform.

Should I use the latest Docker image tag?

The latest tag is convenient but unpredictable. Pinning a tested image version improves repeatability and makes upgrades and rollbacks easier to control.

Are Docker volumes automatically backed up?

No. Docker volumes provide persistent storage, but Docker does not automatically create external backups. Important volumes and databases need a separate, tested backup and restoration process.

The post Docker Issues & Troubleshooting: Real-World Solutions appeared first on John Nessime.