That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that Kubernetes zero-downtime rollouts depend on, and it is not the question that usually bites.

This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.

What a zero-downtime rollout actually requires

Three independent things have to be true at the same time, and each is owned by a different part of the system.

• Capacity never dips below what traffic needs. Owned by the Deployment’s rolling update strategy.
• No pod receives traffic before it can serve it. Owned by the readiness probe.
• No pod receives traffic after it stops serving it. Owned by graceful termination, and this is the one nobody configures.

Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.

Failure family one: traffic arrives before the pod can serve it

This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means “the process was forked”, not “the process can answer HTTP”. The pod lands in the Service’s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.

The readiness probe that lies

Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:

A readiness endpoint that returns 200 from a static handler. If /healthz is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.

A readiness endpoint that checks downstream dependencies. This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks every replica unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers “can this pod serve traffic”, not “is the whole system healthy”. Check the dependency in your app’s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.

Let the startup probe own the boot budget

For anything slow to boot, do not stretch initialDelaySeconds on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.

# The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.

Failure family two: traffic arrives after the pod has stopped

This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has “already added readiness probes”.

When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container’s main process.
2. The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.

Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.

The preStop sleep, and what it does not do

The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.

spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15

If you are on an older cluster without the native sleep handler, the equivalent is an exec hook, which does require a shell and a sleep binary in the image:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application’s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.

There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.

The grace period is a shared budget

This trips people up. The terminationGracePeriodSeconds countdown starts when the pod is marked Terminating, which is before the preStop hook runs, not after. The hook and your application’s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.

Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.

One more detail worth knowing before you go looking for it: probes accept their own terminationGracePeriodSeconds override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.

Failure family three: the rollout deletes capacity faster than it adds it

The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1

Setting maxUnavailable: 0 is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise maxSurge to a percentage if you have the node capacity and want the speed back.

minReadySeconds is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.

Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Failure family four: the load balancer was never watching EndpointSlices

Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.

Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.

Two things help here:

• Pod readiness gates. These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.
• Match your preStop sleep to the deregistration delay. If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.

The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod’s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.

Verifying a zero-downtime rollout instead of hoping

A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.

1. Generate steady traffic against the real ingress path. Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.
2. Trigger a rollout with no image change, so you are testing the mechanism rather than your new code.
3. Watch EndpointSlices in a second terminal and note how long a terminating pod stays in the list.
4. Count non-200 responses. Anything above zero is a bug, not noise.
5. Repeat under realistic concurrency. A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.

# Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m

Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.

Troubleshooting

Rollout hangs, new pods never become Ready. The readiness probe is failing. kubectl describe pod shows the probe failure and the response it got. Check that the port name in the probe matches a declared containerPort name, that the path exists, and that the app is listening on all interfaces rather than only 127.0.0.1, which is a classic one when moving from a local docker-compose setup.

Probes fail only under load. Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within timeoutSeconds, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.

Errors persist after adding a preStop sleep. Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.

Pods stuck Terminating for the full grace period. Your app is ignoring SIGTERM. This is extremely common when the container’s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of ENTRYPOINT, or an init like tini, so your process actually receives the signal.

Rollout succeeded but the new version is broken. Roll back first, investigate second.

# Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3

Common mistakes

• Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.
• Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.
• Leaving maxUnavailable at the default on a service running near capacity.
• Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.
• Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.
• Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.
• Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.

Best practices

• Three separate endpoints: /healthz/started, /healthz/ready, /healthz/live. They answer different questions and should be allowed to disagree.
• maxUnavailable: 0 plus a maxSurge you have node capacity for, as the default for anything user-facing.
• A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.
• A grace period that covers preStop plus your slowest realistic request, with slack.
• Handle SIGTERM properly in the application, and make sure it reaches PID 1.
• A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.
• minReadySeconds long enough to catch a pod that passes its probe and then dies.
• Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.

FAQ

Do readiness probes alone give me zero-downtime deployments?

No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.

How long should the preStop sleep be?

Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.

Should the readiness probe check the database?

Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.

What is the difference between maxSurge and maxUnavailable?

maxSurge is how many pods you may run above the replica count during a rollout. maxUnavailable is how many you may drop below it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.

Why do I still get 502s on EKS, GKE or another managed cluster?

Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group’s deregistration delay rather than against kube-proxy.

Can I get zero-downtime rollouts with a StatefulSet?

The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.

Does kubectl rollout status prove the deploy was clean?

It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.

The one thing worth remembering

Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.

Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.

Need a second pair of eyes on your rollouts?

Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:

• Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change
• Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number
• Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager
• Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths
• Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks
• Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event

If you have a Deployment manifest, a kubectl describe pod output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.

The post Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn’t Saving You appeared first on John Nessime.

That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.

Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.

This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.

What Docker Compose genuinely gets right

Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.

• The whole system fits in one file you can read. Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.
• Dev and prod can share a base file. Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.
• No control plane to operate. A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.
• Recovery is comprehensible. When something breaks at 2am, docker compose ps and docker compose logs tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.

If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.

Failure family one: the deploy gap

This is the one that surprises people who came from a platform that did rolling updates for them.

docker compose up -d is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.

The deploy.update_config block with order: start-first exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.

What to do instead

Three options, in increasing order of effort:

1. Accept the gap and shrink it. Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.
2. Put a reverse proxy in front and run two slots. Traefik, Caddy or plain Nginx in a container, with app-blue and app-green services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.
3. Drain at the edge. If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.

Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:

# Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.

That last line matters more than it looks. A service with no health check is invisible to --wait, so a deploy script can report success while your API is still crash-looping.

Failure family two: the disk fills up and nothing tells you

Docker’s default logging driver is json-file, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local driver is the recommended alternative because it rotates out of the box and uses a more compact format.

So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.

Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}

Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.

You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:

x-logging: &default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    <<: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    <<: *default-logging

Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.

# Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v

Failure family three: state you did not mean to keep

Two related traps here.

The first is anonymous volumes. If an image declares a VOLUME in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.

The second is docker compose down. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.

The practical rule: on a production host, restarting services is docker compose up -d or docker compose restart. down belongs in a maintenance procedure with a fresh backup, not in a daily habit.

Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.

Failure family four: ports that bypass your firewall

This is the one that turns into a security incident rather than an outage.

When you write ports: - "5432:5432", Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status output that looks perfectly reassuring.

The fix is to stop publishing what does not need publishing:

services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]

Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.

While you are in daemon config: set default-address-pools as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.

Failure family five: secrets in environment variables

Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.

The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under /run/secrets/:

services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.

One more thing that trips people up: .env and env_file are different mechanisms. The .env file in the project directory feeds variable interpolation inside the Compose file. env_file passes variables into the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.

Failure family six: health checks that report but never act

A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.

So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.

Health checks are still worth defining, because they gate startup ordering and they gate --wait. Just be clear about what they do not do.

services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s

Note restart: unless-stopped rather than always. The difference shows up after a host reboot: always will start a container you had deliberately stopped, unless-stopped respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.

To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.

Troubleshooting a Compose stack that is misbehaving

The commands I reach for, roughly in order:

# The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' <container>

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"

Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.

Common mistakes

• Deploying :latest. You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.
• Keeping the version: key at the top of the file. Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.
• No resource limits anywhere. One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.
• Running docker compose up from an SSH session and walking away. Without -d the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.
• The Compose file lives only on the server. If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.
• Pulling from Docker Hub anonymously in a deploy script. Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.
• Automatic image updates on production. Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.

Best practices for Docker Compose in production

• One base file plus a production override. Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then docker compose -f compose.yaml -f compose.prod.yaml up -d is your deploy, and the diff between environments is a file you can read.
• Give every service a health check. It gates startup order, it makes --wait meaningful, and it gives you a status worth alerting on.
• Set memory and CPU limits on everything. Watch docker stats under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.
• Segment your networks. An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.
• Back up volumes, and restore one. A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.
• Make the deploy a script, not a memory. Pull, up with --wait, verify, prune. Six lines in the repo beats six commands somebody half-remembers.
• Run docker compose config in CI. It catches a malformed override before it reaches the server, and it costs nothing.

When to stop using Compose

Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.

The signals I treat as a real trigger, rather than as an excuse to rewrite everything:

• You need more than one machine for availability, not just for capacity.
• You need automatic rescheduling when a host dies, without somebody logging in.
• You are scaling components independently and often enough that doing it by hand is a real cost.
• Several teams deploy to the same infrastructure and need isolation from each other.

If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.

Frequently asked questions

Is Docker Compose production ready?

For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.

How do I get zero-downtime deployments with Docker Compose?

Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.

Does docker compose down delete my database?

Not by itself. Plain down removes containers and the project’s networks and leaves named volumes in place. Adding -v removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.

Why does my container keep restarting with exit code 137?

137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats for growth over time and dmesg for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.

Should I still write version: at the top of my Compose file?

No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.

Can I use Docker secrets without Swarm?

Yes, using file-based secrets. Declare a top-level secrets block with a file: source, grant it to a service, and the content appears at /run/secrets/<name> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect.

How many services is too many for one Compose file?

There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.

The one thing worth remembering

Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.

Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.

Need a second pair of eyes on your Compose setup?

I work with teams running containerised workloads on their own servers. Things I am usually brought in for:

• Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first
• Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI
• Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does
• Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall
• Volume backup and restore that has been tested by restoring, not just by running
• An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down

If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.

The post Docker Compose in Production: What Works and What Quietly Burns You appeared first on John Nessime.