The stack had been up for months. Nobody had touched it, nobody had needed to. Then the disk alert fires at an awkward hour, Postgres flips to read-only because it cannot write WAL, and you SSH in to find that one chatty container has written tens of gigabytes into a single JSON log file that nothing was ever going to rotate.<\/p>\n\n\n\n
That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.<\/p>\n\n\n\n
Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.<\/p>\n\n\n\n
This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.<\/p>\n\n\n\n
Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.<\/p>\n\n\n\n
docker compose ps<\/code> and docker compose logs<\/code> tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.<\/li>\n<\/ul>\n\n\n\nIf your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.<\/p>\n\n\n\n
Failure family one: the deploy gap<\/h2>\n\n\n\n
This is the one that surprises people who came from a platform that did rolling updates for them.<\/p>\n\n\n\n
docker compose up -d<\/code> is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.<\/p>\n\n\n\nThe deploy.update_config<\/code> block with order: start-first<\/code> exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.<\/p>\n\n\n\nWhat to do instead<\/h3>\n\n\n\n
Three options, in increasing order of effort:<\/p>\n\n\n\n
\n- Accept the gap and shrink it.<\/strong> Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.<\/li>\n\n
- Put a reverse proxy in front and run two slots.<\/strong> Traefik, Caddy or plain Nginx in a container, with
app-blue<\/code> and app-green<\/code> services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.<\/li>\n\n- Drain at the edge.<\/strong> If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.<\/li>\n<\/ol>\n\n\n\n
Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:<\/p>\n\n\n\n
# Fetch new images while the old containers are still serving traffic.\ndocker compose pull\n\n# Recreate changed services, then block until every service with a\n# healthcheck reports healthy. Non-zero exit if something never gets there.\ndocker compose up -d --wait --wait-timeout 120\n\n# --wait only knows about services that define a healthcheck.\n# Services without one are treated as ready the moment they start.<\/code><\/pre>\n\n\n\nThat last line matters more than it looks. A service with no health check is invisible to --wait<\/code>, so a deploy script can report success while your API is still crash-looping.<\/p>\n\n\n\nFailure family two: the disk fills up and nothing tells you<\/h2>\n\n\n\n
Docker’s default logging driver is json-file<\/code>, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local<\/code> driver is the recommended alternative because it rotates out of the box and uses a more compact format.<\/p>\n\n\n\nSo the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.<\/p>\n\n\n\n
Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:<\/p>\n\n\n\n
\/\/ \/etc\/docker\/daemon.json\n{\n \"log-driver\": \"local\",\n \"log-opts\": {\n \"max-size\": \"10m\",\n \"max-file\": \"3\"\n },\n \"default-address-pools\": [\n { \"base\": \"10.40.0.0\/16\", \"size\": 24 }\n ]\n}<\/code><\/pre>\n\n\n\nTwo things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.<\/p>\n\n\n\n
You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:<\/p>\n\n\n\n
x-logging: &default-logging\n logging:\n driver: local\n options:\n max-size: \"10m\"\n max-file: \"3\"\n\nservices:\n api:\n image: registry.example.com\/api:1.4.2\n <<: *default-logging\n\n worker:\n image: registry.example.com\/worker:1.4.2\n <<: *default-logging<\/code><\/pre>\n\n\n\nLogs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.<\/p>\n\n\n\n
# Images not used by any container and older than a week.\n# -a includes untagged parents, not just dangling layers.\ndocker image prune -a --filter \"until=168h\" --force\n\n# Where the space actually went.\ndocker system df -v<\/code><\/pre>\n\n\n\nFailure family three: state you did not mean to keep<\/h2>\n\n\n\n
Two related traps here.<\/p>\n\n\n\n
The first is anonymous volumes. If an image declares a VOLUME<\/code> in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.<\/p>\n\n\n\nThe second is docker compose down<\/code>. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v<\/code> and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.<\/p>\n\n\n\nThe practical rule: on a production host, restarting services is docker compose up -d<\/code> or docker compose restart<\/code>. down<\/code> belongs in a maintenance procedure with a fresh backup, not in a daily habit.<\/p>\n\n\n\nBind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.<\/p>\n\n\n\n
Failure family four: ports that bypass your firewall<\/h2>\n\n\n\n
This is the one that turns into a security incident rather than an outage.<\/p>\n\n\n\n
When you write ports: - \"5432:5432\"<\/code>, Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status<\/code> output that looks perfectly reassuring.<\/p>\n\n\n\nThe fix is to stop publishing what does not need publishing:<\/p>\n\n\n\n
services:\n db:\n image: postgres:16\n # No ports: block at all. Other services on the same Compose\n # network reach it as db:5432 by service name.\n networks: [backend]\n\n api:\n image: registry.example.com\/api:1.4.2\n # Bound to loopback only. The reverse proxy on this host can\n # reach it; the internet cannot, regardless of firewall state.\n ports:\n - \"127.0.0.1:8080:8080\"\n networks: [backend, edge]<\/code><\/pre>\n\n\n\nTwo habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.<\/p>\n\n\n\n
While you are in daemon config: set default-address-pools<\/code> as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.<\/p>\n\n\n\nFailure family five: secrets in environment variables<\/h2>\n\n\n\n
Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect<\/code>, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.<\/p>\n\n\n\nThe Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under \/run\/secrets\/<\/code>:<\/p>\n\n\n\nservices:\n db:\n image: postgres:16\n environment:\n # Official Postgres, MySQL and Redis images support the _FILE\n # convention: read the value from this path instead of the env var.\n POSTGRES_PASSWORD_FILE: \/run\/secrets\/db_password\n secrets:\n - db_password\n\nsecrets:\n db_password:\n file: .\/secrets\/db_password.txt<\/code><\/pre>\n\n\n\nThe file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect<\/code> output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.<\/p>\n\n\n\nOne more thing that trips people up: .env<\/code> and env_file<\/code> are different mechanisms. The .env<\/code> file in the project directory feeds variable interpolation inside<\/em> the Compose file. env_file<\/code> passes variables into<\/em> the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.<\/p>\n\n\n\nFailure family six: health checks that report but never act<\/h2>\n\n\n\n
A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.<\/p>\n\n\n\n
So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.<\/p>\n\n\n\n
Health checks are still worth defining, because they gate startup ordering and they gate --wait<\/code>. Just be clear about what they do not do.<\/p>\n\n\n\nservices:\n db:\n image: postgres:16\n restart: unless-stopped\n healthcheck:\n # pg_isready exits non-zero until Postgres accepts connections.\n test: [\"CMD-SHELL\", \"pg_isready -U postgres\"]\n interval: 10s\n timeout: 5s\n retries: 5\n # Grace period. Failures during start_period do not count\n # toward retries, so a slow first boot is not marked unhealthy.\n start_period: 30s\n\n api:\n image: registry.example.com\/api:1.4.2\n restart: unless-stopped\n depends_on:\n db:\n # Waits for db to report healthy before creating api.\n # This is a startup-order guarantee only, not a runtime one.\n condition: service_healthy\n healthcheck:\n test: [\"CMD\", \"wget\", \"--spider\", \"-q\", \"http:\/\/localhost:8080\/healthz\"]\n interval: 15s\n timeout: 5s\n retries: 3\n start_period: 20s<\/code><\/pre>\n\n\n\nNote restart: unless-stopped<\/code> rather than always<\/code>. The difference shows up after a host reboot: always<\/code> will start a container you had deliberately stopped, unless-stopped<\/code> respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.<\/p>\n\n\n\nTo close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.<\/p>\n\n\n\n
Troubleshooting a Compose stack that is misbehaving<\/h2>\n\n\n\n
The commands I reach for, roughly in order:<\/p>\n\n\n\n
# The fully resolved config: overrides merged, variables substituted,\n# anchors expanded. This is what Compose is actually going to run.\n# Careful, it prints secret VALUES from interpolation.\ndocker compose config\n\n# Health state and exit codes, not just up\/down.\ndocker compose ps --all\n\n# Why did it die? Look at the last lines before the restart.\ndocker compose logs --tail=200 --timestamps api\n\n# The health check's own output, which is where the real error usually is.\ndocker inspect --format '{{json .State.Health}}' <container>\n\n# Live resource use. Sudden memory growth before a restart means OOM.\ndocker stats --no-stream\n\n# Did the kernel OOM-killer take it? Exit code 137 is the hint.\ndmesg --ctime | grep -i \"out of memory\"<\/code><\/pre>\n\n\n\nTwo patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.<\/p>\n\n\n\n
Common mistakes<\/h2>\n\n\n\n\n- Deploying
:latest<\/code>.<\/strong> You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.<\/li>\n\n- Keeping the
version:<\/code> key at the top of the file.<\/strong> Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.<\/li>\n\n- No resource limits anywhere.<\/strong> One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.<\/li>\n\n
- Running
docker compose up<\/code> from an SSH session and walking away.<\/strong> Without -d<\/code> the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.<\/li>\n\n- The Compose file lives only on the server.<\/strong> If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.<\/li>\n\n
- Pulling from Docker Hub anonymously in a deploy script.<\/strong> Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.<\/li>\n\n
- Automatic image updates on production.<\/strong> Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.<\/li>\n<\/ul>\n\n\n\n
Best practices for Docker Compose in production<\/h2>\n\n\n\n\n- One base file plus a production override.<\/strong> Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then
docker compose -f compose.yaml -f compose.prod.yaml up -d<\/code> is your deploy, and the diff between environments is a file you can read.<\/li>\n\n- Give every service a health check.<\/strong> It gates startup order, it makes
--wait<\/code> meaningful, and it gives you a status worth alerting on.<\/li>\n\n- Set memory and CPU limits on everything.<\/strong> Watch
docker stats<\/code> under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.<\/li>\n\n- Segment your networks.<\/strong> An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.<\/li>\n\n
- Back up volumes, and restore one.<\/strong> A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.<\/li>\n\n
- Make the deploy a script, not a memory.<\/strong> Pull, up with
--wait<\/code>, verify, prune. Six lines in the repo beats six commands somebody half-remembers.<\/li>\n\n- Run
docker compose config<\/code> in CI.<\/strong> It catches a malformed override before it reaches the server, and it costs nothing.<\/li>\n<\/ul>\n\n\n\nWhen to stop using Compose<\/h2>\n\n\n\n
Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.<\/p>\n\n\n\n
The signals I treat as a real trigger, rather than as an excuse to rewrite everything:<\/p>\n\n\n\n
\n- You need more than one machine for availability, not just for capacity.<\/li>\n\n
- You need automatic rescheduling when a host dies, without somebody logging in.<\/li>\n\n
- You are scaling components independently and often enough that doing it by hand is a real cost.<\/li>\n\n
- Several teams deploy to the same infrastructure and need isolation from each other.<\/li>\n<\/ul>\n\n\n\n
If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.<\/p>\n\n\n\n
\n\n\n\nFrequently asked questions<\/h2>\n\n\n\nIs Docker Compose production ready?<\/h3>\n\n\n\n
For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.<\/p>\n\n\n\n
How do I get zero-downtime deployments with Docker Compose?<\/h3>\n\n\n\n
Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.<\/p>\n\n\n\n
Does docker compose down<\/code> delete my database?<\/h3>\n\n\n\nNot by itself. Plain down<\/code> removes containers and the project’s networks and leaves named volumes in place. Adding -v<\/code> removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.<\/p>\n\n\n\nWhy does my container keep restarting with exit code 137?<\/h3>\n\n\n\n
137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats<\/code> for growth over time and dmesg<\/code> for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.<\/p>\n\n\n\nShould I still write version:<\/code> at the top of my Compose file?<\/h3>\n\n\n\nNo. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.<\/p>\n\n\n\n
Can I use Docker secrets without Swarm?<\/h3>\n\n\n\n
Yes, using file-based secrets. Declare a top-level secrets<\/code> block with a file:<\/code> source, grant it to a service, and the content appears at \/run\/secrets\/<name><\/code> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect<\/code>.<\/p>\n\n\n\nHow many services is too many for one Compose file?<\/h3>\n\n\n\n
There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.<\/p>\n\n\n\n
The one thing worth remembering<\/h2>\n\n\n\n
Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.<\/p>\n\n\n\n
Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down<\/code>, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.<\/p>\n\n\n\n
\n\n\n\nNeed a second pair of eyes on your Compose setup?<\/h2>\n\n\n\n
I work with teams running containerised workloads on their own servers. Things I am usually brought in for:<\/p>\n\n\n\n
\n- Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first<\/li>\n\n
- Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI<\/li>\n\n
- Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does<\/li>\n\n
- Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall<\/li>\n\n
- Volume backup and restore that has been tested by restoring, not just by running<\/li>\n\n
- An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down<\/li>\n<\/ul>\n\n\n\n
If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all<\/code>, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.<\/p>\n\n\n\n