That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.

That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.

This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn’t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.

Why the exit path matters more than the launch

All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.

Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.

So the question is not “which is cheapest for a small site.” It is “how much of this stack is portable, and who operates it after handover.”

Amazon Lightsail: where it wins and where it doesn’t

Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.

Where it wins

• Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.
• The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.
• Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.
• IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.

Where it doesn’t

• Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the default VPC in the same account and the same region. Not a custom VPC. Not another account.
• Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.
• The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.
• No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.

Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail’s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.

Amazon EC2: where it wins and where it doesn’t

EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.

Where it wins

• Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.
• Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.
• Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.
• Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.

Where it doesn’t

• Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.
• Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.
• Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.
• You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.

The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.

Amazon ECS: where it wins and where it doesn’t

ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.

• ECS on Fargate bills per vCPU-hour and GB-hour of the resources you declare in the task definition, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.
• ECS on EC2 means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.

Where it wins

• The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.
• Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.
• Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.
• Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.
• If the app is already containerized and CI already builds images, you are most of the way there.

Where it doesn’t

• The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.
• Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.
• Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.
• Handover is harder. A client’s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.

The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.

# Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"

Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.

Lightsail vs EC2 vs ECS on the lines that move the bill

Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.

• Egress. Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.
• NAT gateway. Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.
• Load balancing. Lightsail’s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.
• Public IPv4. Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.
• Log ingestion. CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.
• Idle resources. Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.

A decision procedure for Lightsail vs EC2 vs ECS

Run these in order. The first one that gives a clear answer is the answer.

1. Does this workload need to talk privately to other AWS resources? If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.
2. Who operates this in a year? If the answer is “the client’s office manager” or “nobody,” bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.
3. Is the app already containerized, with CI that builds images? If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.
4. Is the load steady or spiky? Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.
5. How much egress? Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail’s bundled allowance is a real argument.
6. What is the audit story? If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.

If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works before you need it. The commands are short:

# Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records

Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.

If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:

aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1

You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.

Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.

Arguments that don’t survive contact

“Lightsail is just EC2 with a nicer console”

Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.

“Start on Lightsail, migrate to EC2 when you outgrow it”

Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.

“Fargate is serverless, so it scales to zero and costs nothing when idle”

The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word “serverless” gives you.

“Containers are better practice, so ECS is the professional choice”

Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.

“Reserved Instances make EC2 cheaper, so always pick EC2”

Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client’s behalf is a conversation, not a default.

FAQ

Is Lightsail cheaper than EC2 for a small website?

Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.

Can a Lightsail instance connect to an RDS database?

Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the default VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.

Does everything migrate when I export Lightsail to EC2?

No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel & WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.

Is ECS overkill for a single small container?

Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.

Why is my Fargate bill higher than the vCPU-hour rate suggests?

Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.

Should I use Lightsail container service instead of ECS?

Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.

The one thing worth remembering

Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.

One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.

Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.

Need a second opinion before you commit?

Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:

• Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses
• Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency
• Planning and executing Lightsail to EC2 migrations, including the parts that do not export
• Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it
• Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy
• Writing the handover runbook so the next person can operate it without calling you

If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.

The post Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads appeared first on John Nessime.

That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that Kubernetes zero-downtime rollouts depend on, and it is not the question that usually bites.

This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.

What a zero-downtime rollout actually requires

Three independent things have to be true at the same time, and each is owned by a different part of the system.

• Capacity never dips below what traffic needs. Owned by the Deployment’s rolling update strategy.
• No pod receives traffic before it can serve it. Owned by the readiness probe.
• No pod receives traffic after it stops serving it. Owned by graceful termination, and this is the one nobody configures.

Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.

Failure family one: traffic arrives before the pod can serve it

This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means “the process was forked”, not “the process can answer HTTP”. The pod lands in the Service’s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.

The readiness probe that lies

Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:

A readiness endpoint that returns 200 from a static handler. If /healthz is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.

A readiness endpoint that checks downstream dependencies. This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks every replica unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers “can this pod serve traffic”, not “is the whole system healthy”. Check the dependency in your app’s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.

Let the startup probe own the boot budget

For anything slow to boot, do not stretch initialDelaySeconds on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.

# The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.

Failure family two: traffic arrives after the pod has stopped

This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has “already added readiness probes”.

When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container’s main process.
2. The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.

Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.

The preStop sleep, and what it does not do

The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.

spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15

If you are on an older cluster without the native sleep handler, the equivalent is an exec hook, which does require a shell and a sleep binary in the image:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application’s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.

There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.

The grace period is a shared budget

This trips people up. The terminationGracePeriodSeconds countdown starts when the pod is marked Terminating, which is before the preStop hook runs, not after. The hook and your application’s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.

Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.

One more detail worth knowing before you go looking for it: probes accept their own terminationGracePeriodSeconds override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.

Failure family three: the rollout deletes capacity faster than it adds it

The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1

Setting maxUnavailable: 0 is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise maxSurge to a percentage if you have the node capacity and want the speed back.

minReadySeconds is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.

Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Failure family four: the load balancer was never watching EndpointSlices

Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.

Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.

Two things help here:

• Pod readiness gates. These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.
• Match your preStop sleep to the deregistration delay. If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.

The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod’s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.

Verifying a zero-downtime rollout instead of hoping

A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.

1. Generate steady traffic against the real ingress path. Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.
2. Trigger a rollout with no image change, so you are testing the mechanism rather than your new code.
3. Watch EndpointSlices in a second terminal and note how long a terminating pod stays in the list.
4. Count non-200 responses. Anything above zero is a bug, not noise.
5. Repeat under realistic concurrency. A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.

# Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m

Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.

Troubleshooting

Rollout hangs, new pods never become Ready. The readiness probe is failing. kubectl describe pod shows the probe failure and the response it got. Check that the port name in the probe matches a declared containerPort name, that the path exists, and that the app is listening on all interfaces rather than only 127.0.0.1, which is a classic one when moving from a local docker-compose setup.

Probes fail only under load. Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within timeoutSeconds, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.

Errors persist after adding a preStop sleep. Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.

Pods stuck Terminating for the full grace period. Your app is ignoring SIGTERM. This is extremely common when the container’s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of ENTRYPOINT, or an init like tini, so your process actually receives the signal.

Rollout succeeded but the new version is broken. Roll back first, investigate second.

# Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3

Common mistakes

• Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.
• Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.
• Leaving maxUnavailable at the default on a service running near capacity.
• Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.
• Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.
• Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.
• Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.

Best practices

• Three separate endpoints: /healthz/started, /healthz/ready, /healthz/live. They answer different questions and should be allowed to disagree.
• maxUnavailable: 0 plus a maxSurge you have node capacity for, as the default for anything user-facing.
• A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.
• A grace period that covers preStop plus your slowest realistic request, with slack.
• Handle SIGTERM properly in the application, and make sure it reaches PID 1.
• A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.
• minReadySeconds long enough to catch a pod that passes its probe and then dies.
• Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.

FAQ

Do readiness probes alone give me zero-downtime deployments?

No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.

How long should the preStop sleep be?

Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.

Should the readiness probe check the database?

Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.

What is the difference between maxSurge and maxUnavailable?

maxSurge is how many pods you may run above the replica count during a rollout. maxUnavailable is how many you may drop below it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.

Why do I still get 502s on EKS, GKE or another managed cluster?

Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group’s deregistration delay rather than against kube-proxy.

Can I get zero-downtime rollouts with a StatefulSet?

The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.

Does kubectl rollout status prove the deploy was clean?

It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.

The one thing worth remembering

Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.

Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.

Need a second pair of eyes on your rollouts?

Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:

• Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change
• Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number
• Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager
• Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths
• Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks
• Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event

If you have a Deployment manifest, a kubectl describe pod output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.

The post Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn’t Saving You appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.

That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.

Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.

This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.

What Docker Compose genuinely gets right

Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.

• The whole system fits in one file you can read. Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.
• Dev and prod can share a base file. Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.
• No control plane to operate. A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.
• Recovery is comprehensible. When something breaks at 2am, docker compose ps and docker compose logs tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.

If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.

Failure family one: the deploy gap

This is the one that surprises people who came from a platform that did rolling updates for them.

docker compose up -d is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.

The deploy.update_config block with order: start-first exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.

What to do instead

Three options, in increasing order of effort:

1. Accept the gap and shrink it. Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.
2. Put a reverse proxy in front and run two slots. Traefik, Caddy or plain Nginx in a container, with app-blue and app-green services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.
3. Drain at the edge. If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.

Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:

# Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.

That last line matters more than it looks. A service with no health check is invisible to --wait, so a deploy script can report success while your API is still crash-looping.

Failure family two: the disk fills up and nothing tells you

Docker’s default logging driver is json-file, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local driver is the recommended alternative because it rotates out of the box and uses a more compact format.

So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.

Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}

Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.

You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:

x-logging: &default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    <<: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    <<: *default-logging

Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.

# Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v

Failure family three: state you did not mean to keep

Two related traps here.

The first is anonymous volumes. If an image declares a VOLUME in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.

The second is docker compose down. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.

The practical rule: on a production host, restarting services is docker compose up -d or docker compose restart. down belongs in a maintenance procedure with a fresh backup, not in a daily habit.

Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.

Failure family four: ports that bypass your firewall

This is the one that turns into a security incident rather than an outage.

When you write ports: - "5432:5432", Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status output that looks perfectly reassuring.

The fix is to stop publishing what does not need publishing:

services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]

Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.

While you are in daemon config: set default-address-pools as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.

Failure family five: secrets in environment variables

Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.

The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under /run/secrets/:

services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.

One more thing that trips people up: .env and env_file are different mechanisms. The .env file in the project directory feeds variable interpolation inside the Compose file. env_file passes variables into the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.

Failure family six: health checks that report but never act

A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.

So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.

Health checks are still worth defining, because they gate startup ordering and they gate --wait. Just be clear about what they do not do.

services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s

Note restart: unless-stopped rather than always. The difference shows up after a host reboot: always will start a container you had deliberately stopped, unless-stopped respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.

To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.

Troubleshooting a Compose stack that is misbehaving

The commands I reach for, roughly in order:

# The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' <container>

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"

Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.

Common mistakes

• Deploying :latest. You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.
• Keeping the version: key at the top of the file. Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.
• No resource limits anywhere. One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.
• Running docker compose up from an SSH session and walking away. Without -d the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.
• The Compose file lives only on the server. If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.
• Pulling from Docker Hub anonymously in a deploy script. Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.
• Automatic image updates on production. Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.

Best practices for Docker Compose in production

• One base file plus a production override. Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then docker compose -f compose.yaml -f compose.prod.yaml up -d is your deploy, and the diff between environments is a file you can read.
• Give every service a health check. It gates startup order, it makes --wait meaningful, and it gives you a status worth alerting on.
• Set memory and CPU limits on everything. Watch docker stats under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.
• Segment your networks. An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.
• Back up volumes, and restore one. A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.
• Make the deploy a script, not a memory. Pull, up with --wait, verify, prune. Six lines in the repo beats six commands somebody half-remembers.
• Run docker compose config in CI. It catches a malformed override before it reaches the server, and it costs nothing.

When to stop using Compose

Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.

The signals I treat as a real trigger, rather than as an excuse to rewrite everything:

• You need more than one machine for availability, not just for capacity.
• You need automatic rescheduling when a host dies, without somebody logging in.
• You are scaling components independently and often enough that doing it by hand is a real cost.
• Several teams deploy to the same infrastructure and need isolation from each other.

If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.

Frequently asked questions

Is Docker Compose production ready?

For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.

How do I get zero-downtime deployments with Docker Compose?

Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.

Does docker compose down delete my database?

Not by itself. Plain down removes containers and the project’s networks and leaves named volumes in place. Adding -v removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.

Why does my container keep restarting with exit code 137?

137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats for growth over time and dmesg for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.

Should I still write version: at the top of my Compose file?

No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.

Can I use Docker secrets without Swarm?

Yes, using file-based secrets. Declare a top-level secrets block with a file: source, grant it to a service, and the content appears at /run/secrets/<name> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect.

How many services is too many for one Compose file?

There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.

The one thing worth remembering

Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.

Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.

Need a second pair of eyes on your Compose setup?

I work with teams running containerised workloads on their own servers. Things I am usually brought in for:

• Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first
• Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI
• Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does
• Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall
• Volume backup and restore that has been tested by restoring, not just by running
• An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down

If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.

The post Docker Compose in Production: What Works and What Quietly Burns You appeared first on John Nessime.

Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was Delete, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.

That is the sharp edge of Kubernetes persistent volumes, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.

This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.

Three objects, one disk

The mental model that prevents most of this:

• StorageClass is the recipe. It says which driver provisions volumes and what the defaults are.
• PersistentVolume is the actual piece of storage in the cluster. Usually created for you rather than by you.
• PersistentVolumeClaim is the request a pod makes, and the thing that binds one-to-one with a PV.

The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:

# Read the RECLAIMPOLICY column. On managed clusters it usually
# says Delete, and nothing anywhere warns you about that.
kubectl get storageclass

For anything holding real data, define your own class with Retain and use it deliberately:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain              # PV outlives the PVC
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3

The trade-off is real and worth stating: Retain means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.

Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:

kubectl patch pv "$PV_NAME" 
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

ReadWriteOnce does not mean one pod

It means one node. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.

For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:

spec:
  accessModes:
    - ReadWriteOncePod        # enforced by the scheduler

The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.

spec:
  strategy:
    type: Recreate            # old pod goes first, then the new one

That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.

Volumes pin pods to a zone

Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in Pending after a node replacement or a scale-up that happened to land elsewhere.

kubectl describe pod tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.

The preventive setting is volumeBindingMode: WaitForFirstConsumer, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.

StatefulSet or Deployment

The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same volume, which is only correct when you have one replica.

So: one instance with a disk can be a Deployment with Recreate. Anything that scales and keeps per-replica state should be a StatefulSet.

StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to Retain:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain       # deleting the StatefulSet keeps the data
    whenScaled: Delete        # scaling down cleans up the extra replicas

That pairing is the sensible default for most workloads. Be careful with whenScaled: Delete on anything where you scale down temporarily and expect the data back when you scale up again.

Two one-way doors

Volumes grow, they do not shrink. Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.

A volume snapshot is not a database backup. Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.

Troubleshooting

Pod stuck in Pending. Read the events at the bottom of kubectl describe pod. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.

PVC stuck in Pending. kubectl describe pvc names the provisioner error. With WaitForFirstConsumer this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.

Rollout hangs forever on a stateful app. ReadWriteOnce plus rolling update. Switch to Recreate.

PVC will not delete. A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.

Volume is full but the PVC says it has room. You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.

Common mistakes

• Running production data on the default StorageClass without checking its reclaim policy.
• Assuming ReadWriteOnce prevents two pods from writing.
• A ReadWriteOnce volume on a Deployment with rolling updates.
• Scaling a Deployment with a PVC past one replica.
• Leaving volumeBindingMode: Immediate, then debugging Pending pods for an hour.
• Treating volume snapshots as database backups.
• Over-sizing a PVC on the assumption you can shrink it later.
• Removing a stuck finalizer instead of finding what is holding the volume.
• Deleting a namespace without checking what claims live in it.

Best practices

• Your own StorageClass with Retain for anything you would miss, and the default for everything else.
• WaitForFirstConsumer on every class you define.
• allowVolumeExpansion: true from the start, because adding it later does not help existing volumes.
• ReadWriteOncePod where exactly one writer is a correctness requirement.
• StatefulSets for per-replica state, single-replica Deployments with Recreate for everything else.
• An explicit persistentVolumeClaimRetentionPolicy rather than relying on the default.
• Application-level backups for databases, tested by restoring one.
• Alerts on volume capacity, because a full disk is the most predictable outage there is.

FAQ

Does deleting a PVC delete my data?

If the reclaim policy is Delete, yes, and that is the default nearly everywhere. Run kubectl get storageclass and read the RECLAIMPOLICY column before you assume otherwise.

Can two pods share a ReadWriteOnce volume?

Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.

Why is my pod Pending with a bound PVC?

Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in kubectl describe pod say so. WaitForFirstConsumer prevents it happening again.

Should I run databases in Kubernetes at all?

You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.

Can I shrink a PersistentVolumeClaim?

No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.

The one thing to remember

Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.

Go and run kubectl get storageclass on your production cluster now. If the reclaim policy on the class holding your database says Delete, you have found something worth fixing before lunch.

Need a hand with stateful workloads?

Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:

• Auditing a cluster’s storage classes, reclaim policies and access modes, and telling you what a stray kubectl delete would actually destroy.
• Moving stateful applications onto StatefulSets with sane retention, expansion and scheduling behaviour.
• Backup and restore design for data in Kubernetes, including an actual restore test rather than a documented intention.
• Diagnosing Pending pods, stuck rollouts and volumes that will not detach.
• CSI driver setup and storage class design on EKS and self-managed clusters.
• Capacity alerting so a full volume is a notification instead of an incident.

Send me the output of kubectl get storageclass and kubectl get pv, and I will tell you what stands out.

The post It’s Only a Claim: Kubernetes Persistent Volumes and Stateful Workloads appeared first on John Nessime.

The application behind all of this is one Go binary and a Postgres database. It serves a few hundred requests a minute. Nobody has touched its code in a fortnight.

That is the moment worth noticing: when Kubernetes is overkill, the tell is not that anything is broken. Everything works. It is that the cluster has quietly become the thing you operate, and the product has become a tenant.

What it actually buys you

Worth being fair, because the case for Kubernetes is real and I reach for it regularly.

It gives you bin-packing across many workloads, so a fleet of services shares machines efficiently. It gives you a declarative API that a platform team can build on, so twelve product teams deploy without twelve bespoke pipelines. It gives you self-healing, rolling updates, horizontal autoscaling and namespace-level isolation as defaults rather than as things you script. And it gives you a portable vocabulary: an engineer who knows Kubernetes knows most of your infrastructure on day one.

Every one of those benefits scales with the number of services and the number of teams. None of them scale with how modern you want to feel.

The bill nobody itemises

The cost is not the control plane fee. It is the surface area you have agreed to keep current.

Kubernetes ships three minor releases a year and supports each for about fourteen months, and you can skip at most one minor version when upgrading. So there is a permanent, non-negotiable upgrade cadence: roughly every four months something needs planning, testing and a maintenance window, forever. Fall behind and you are doing multi-hop upgrades on an unsupported version, which is the worst combination available.

Underneath that sit the components nobody counts when they say “we’ll just run Kubernetes”: a CNI plugin, a CSI driver, an ingress controller, cert-manager, external-dns, a metrics pipeline, an autoscaler, RBAC, and whatever operators your databases need. Each has its own release cycle, its own breaking changes and its own opinions about the others. That is where the Tuesday goes.

Then there is debugging. A request that used to fail in one place now fails in one of several, and telling apart an application bug from a NetworkPolicy, a readiness probe, a DNS caching issue or a node under memory pressure is a genuine skill. On a good team that skill exists. On a three-person team it exists in one person’s head and goes on holiday with them.

Signals you don’t need it yet

• You have fewer than about five services, and most of them are one binary and a database.
• One team owns everything, so there is nothing to isolate from anyone.
• Traffic is predictable enough that you size for peak and stop thinking about it.
• Nobody on the team has run a cluster in production before.
• Your deploys are already fine. Nobody is complaining about them.
• You cannot name the specific problem Kubernetes is solving, only the general one.
• The cluster would run on two nodes, which means you have a scheduler for a decision with two possible answers.

Signals you probably do

• Multiple teams deploying independently and tripping over each other’s environments.
• Enough services that per-service deployment scripts have become their own maintenance problem.
• Genuinely variable load where autoscaling saves real money, not hypothetical money.
• A compliance or tenancy requirement that maps cleanly onto namespaces and network policy.
• You are already running an operator-based product, like a database platform, that expects Kubernetes.
• Somebody’s whole job is the platform, and it is not also somebody’s whole job to ship features.

That last one is the honest test. Kubernetes is not expensive to install. It is expensive to own, and ownership needs a name attached to it.

What I’d reach for instead

Roughly in order of how much you take on.

One server, systemd, and a reverse proxy. Underrated to the point of being contrarian, and completely adequate for a large number of real businesses. A VPS from Hetzner, InterServer or DigitalOcean, nginx or Caddy in front, systemd units for your services, automated backups off the box. Restart policies and health checks are built in; you are not going without them, you are just getting them from init instead of from a control loop.

Docker Compose on one box. The step up when you want containers without a scheduler. The entire deployment is a file you can read in one screen:

services:
  app:
    image: ghcr.io/acme/api:1.4.2      # pinned, deployed by changing this line
    restart: always
    ports: ["127.0.0.1:8080:8080"]     # proxy handles TLS and the public port
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16                 # pin the major, upgrade deliberately
    restart: always
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes:
  pgdata:

The honest limitation: one box is one box. Losing it is an outage, and a redeploy is a brief gap unless you put something in front. For plenty of internal tools and early products, that is an acceptable trade stated out loud rather than an oversight.

A platform that runs containers for you. Cloud Run, App Runner, Fly.io, Render, Railway. You hand over a container image and get scaling, TLS, rolling deploys and health checks without operating any of it. This is the option most teams should look at hardest, because it buys back the majority of what Kubernetes offers at almost none of the operational cost. You pay in per-unit pricing and in reduced control, and both are usually worth it below a certain scale.

ECS on Fargate, or Nomad. Real schedulers with a fraction of the concepts. If you need orchestration but not the ecosystem, these are legitimate destinations rather than waypoints on the road to Kubernetes.

And if you have decided Kubernetes genuinely is the answer, use a managed control plane, and consider a lightweight distribution such as k3s for small or edge deployments. Do not build the cluster yourself to save money. That is the most expensive saving in infrastructure.

Arguments that don’t survive contact

“We’ll need it eventually.” Possibly. Adopting it three years early costs three years of upgrade cycles to buy an option you may not exercise, and the migration is not much harder later than it is now.

“It’s the industry standard.” For orchestrating fleets, yes. Most applications are not fleets. Standard does not mean universally appropriate, and the same argument would put a load balancer in front of a single server.

“It avoids vendor lock-in.” Partly true and frequently oversold. You have swapped a cloud provider’s API for a distribution, a set of controllers, and a pile of YAML that encodes assumptions about all of them. Portability lives in your container images and your data, not in the scheduler.

“It’ll help us hire.” It helps you hire people who want to run Kubernetes. Whether that is the same as the people who will build your product is worth a moment’s thought.

You’re not painting yourself into a corner

This is the part that makes the decision easy to defer, and it is the most useful thing in this post.

The durable investments are containerising your application, externalising configuration into environment variables, keeping state out of the application and in a database or object storage, exposing a health endpoint, and logging to stdout. Do those and your app already satisfies most of what a Kubernetes manifest asks for. The scheduler underneath is genuinely swappable.

So the choice is not Kubernetes now or a painful rewrite later. It is Kubernetes now, or the same containers running somewhere simpler until the complexity is earned.

FAQ

How many services before Kubernetes makes sense?

There is no clean number, and anyone offering one is guessing. The better question is how many teams deploy independently. One team can coordinate without a platform. Four cannot, and that is when the abstraction starts paying for itself.

Is Docker Compose really production-ready?

On a single host, for a workload that tolerates brief restarts, yes. It runs plenty of real systems. What it does not give you is failover across machines, so be clear that losing the host means an outage and decide whether that is acceptable rather than discovering it.

Doesn’t managed Kubernetes remove the operational burden?

It removes the control plane, which is the part that was never your main problem. Ingress, storage, networking, upgrades and the add-on ecosystem all remain yours. Managed offerings genuinely help, and they narrow the gap rather than closing it.

What about autoscaling? I can’t get that elsewhere.

You can. Cloud Run, App Runner, Fly and ECS all autoscale, several of them to zero, with no cluster to maintain. Autoscaling is a common reason to reach for Kubernetes and one of the weakest, because it is the feature most thoroughly commoditised elsewhere.

How hard is it to migrate later?

Much easier than migrating off it. If your app is containerised, stateless, configured by environment and observable, moving to Kubernetes is mostly writing manifests. The hard parts of that migration are the ones you should be fixing anyway.

The one thing to remember

Kubernetes is not complicated by accident. It is complicated because it solves a genuinely complicated problem, and if you do not have that problem you are paying the complexity and collecting none of the benefit.

So make it a decision with a reason attached. Write down the specific thing you cannot do today, and check whether a cluster is the cheapest way to fix it. Sometimes it is. Often the honest answer is one server, a reverse proxy, and an afternoon spent on backups instead.

Trying to decide?

I get called in on both sides of this: teams drowning in a cluster they did not need, and teams who genuinely need one and are putting it off. Work I take on:

• A straight assessment of whether Kubernetes fits your team and workload, with the reasoning written down rather than an opinion delivered.
• Building the simpler thing properly: single-host or Compose deployments with TLS, health checks, deploys and backups that hold up in production.
• Moving off an over-provisioned cluster onto something proportionate, without downtime.
• Getting an application genuinely portable so the decision stays reversible: containerised, stateless, configured by environment, observable.
• Setting up managed Kubernetes properly when it is the right answer, including the upgrade process nobody plans for.
• Reviewing the cost, in money and in engineering hours, of what you currently run.

Tell me how many services you run and how many people deploy them, and I will tell you what I would build.

The post The Cluster Becomes the Product: When Kubernetes Is Overkill appeared first on John Nessime.

Docker disk usage grows quietly because every part of it accumulates independently — images, stopped containers, volumes, build cache, and container logs, which are the one nobody expects and which no Docker command reports on. Cleaning it up is easy. Cleaning it up without destroying something takes about five more minutes.

This is the order I work through it: measure first, reclaim from safest to most destructive, then fix the thing that caused the growth so you are not back here next month.

Measure before you delete

Every cleanup starts here, and skipping it is how people delete the wrong thing:

docker system df

That gives you four rows — Images, Containers, Local Volumes, Build Cache — with a total size and a reclaimable figure for each. The reclaimable column is the one that matters. If build cache is eleven gigabytes and all of it is reclaimable, you have found your win and you never need to go near volumes.

For the itemised version:

docker system df -v

Now you can see individual images with their sizes, which containers are using what, and which volumes have no container attached. Read this before running anything destructive.

Two more views worth having open:

# What is actually running, and what is just sitting there stopped
docker ps -a --format 'table {{.Names}}t{{.Image}}t{{.Status}}t{{.Size}}'

# Images by age, oldest first
docker images --format 'table {{.Repository}}:{{.Tag}}t{{.Size}}t{{.CreatedSince}}'

Reclaim in order, safest first

These are ordered deliberately. Work down the list and stop as soon as you have enough space. Most of the time you never reach step four.

1. Stopped containers

# See what would go
docker ps -a --filter status=exited --filter status=created

docker container prune

This removes stopped containers and their writable layers. Anything running is untouched, and named volumes attached to those containers survive — only the container’s own layer goes.

The one caveat: if a stopped container held state in its writable layer rather than in a volume, that state is gone. That is a design problem rather than a cleanup problem, but it is worth a glance before you confirm.

2. Build cache

On any machine that builds images regularly, this is usually the biggest single win and it is completely safe. Build cache is derived data. Deleting it costs you a slower next build and nothing else.

# Everything older than a week
docker builder prune --filter "until=168h"

# Everything, no exceptions
docker builder prune -a

3. Dangling images

# Look first
docker images -f dangling=true

docker image prune

Without -a, this removes only dangling images — untagged layers left behind when a rebuild moved a tag to a new image. They are almost always genuine garbage.

4. Unused images — read this one carefully

docker image prune -a

Adding -a changes the meaning completely. It removes every image that does not have at least one container associated with it — running or stopped. So an image you pulled for a job that runs weekly, with no container currently existing, is gone. It will pull again, which is fine if you have bandwidth and the registry is reachable, and much less fine at 3am on a machine with a slow link.

Safer in most cases is an age filter, which keeps anything recent:

docker image prune -a --filter "until=336h"   # older than 14 days

5. Volumes — the one that loses data

Volumes are where databases live. Treat this step as a data operation, not a cleanup.

Before removing anything, look inside the candidates. This mounts each dangling volume read-only into a throwaway container so you can see what is actually in it:

docker volume ls -qf dangling=true | while read -r v; do
  echo "=== $v"
  docker run --rm -v "$v":/vol:ro alpine sh -c 'du -sh /vol; ls -la /vol' 2>/dev/null | head -12
done

If you see anything resembling a data directory, back it up before it goes:

docker run --rm 
  -v my-volume:/vol:ro 
  -v "$PWD":/backup 
  alpine tar czf /backup/my-volume.tar.gz -C /vol .

Only then:

docker volume prune

On recent Docker versions this prunes only anonymous volumes by default, with --all needed to include named ones — a genuinely good safety change. Older versions removed named volumes too. Check which behaviour you have before trusting it:

docker version --format '{{.Server.Version}}'
docker volume prune --help

The gigabytes Docker never tells you about

Here is the part that catches experienced people. Container logs do not appear in docker system df at all, and no prune command touches them. A chatty application with the default logging driver will happily write tens of gigabytes and nothing in Docker’s own tooling will mention it.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -10

To map a log file back to the container that owns it:

docker ps -aq | xargs docker inspect --format '{{.Name}} {{.LogPath}}'

Truncate, do not delete

This matters. Deleting a log file that a running container still has open removes the directory entry but not the data — the kernel keeps the blocks allocated until the process closes the descriptor, so df shows no improvement and you are left confused. Truncate instead:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Then stop it happening again

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

One thing people get wrong here: this applies to newly created containers only. Anything already running keeps the logging config it was created with. You have to recreate those containers for the limit to take effect — restarting them is not enough.

While you are in that file, cap the build cache too:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

Automating it without automating a disaster

A weekly cron is worth having, provided it only ever does the safe operations. The rule is simple: never put --volumes in anything unattended.

# /etc/cron.d/docker-prune
# Weekly, Sunday 03:00. Containers, networks, dangling images and build
# cache older than seven days. No volumes, ever.
0 3 * * 0 root /usr/bin/docker system prune -f --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Note what docker system prune covers without flags: stopped containers, unused networks, dangling images and build cache. It does not touch volumes unless you explicitly pass --volumes, and it does not remove tagged images unless you pass -a. Those two flags are the whole difference between a routine job and an incident.

If you are tempted to add --volumes to a cron job so you do not have to think about it again, you have swapped a disk space problem for a data loss problem on a timer.

Troubleshooting

Pruned everything, df has not moved

Deleted files still held open by a running process. Find them:

sudo lsof +L1 | head -20

Restarting the process that holds the descriptor releases the blocks. If it is the Docker daemon itself, a systemctl restart docker does it, at the cost of bouncing every container without a restart policy.

“No space left on device” but df shows free space

You are out of inodes rather than bytes. Docker’s layered storage creates enormous numbers of small files:

df -i

The fix is the same — prune images and build cache, which is where the file count lives.

overlay2 is huge and I have hardly any images

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2 2>/dev/null

Usually build cache or container writable layers rather than images. Do not delete anything under overlay2 by hand — the directory names are referenced in Docker’s metadata, and removing them directly corrupts the daemon’s view of the world. Always go through docker commands.

Running low and need space right now

Fastest safe sequence, in order:

docker builder prune -a -f
docker container prune -f
docker image prune -f
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Four commands, no volumes touched, no tagged images removed. That resolves the large majority of Docker disk usage emergencies.

Common mistakes

• Reaching for docker system prune -a --volumes first because it is the top search result.
• Putting --volumes in a cron job.
• Deleting log files instead of truncating them, then wondering why df did not change.
• Setting log limits in daemon.json and assuming existing containers pick them up. They do not.
• Removing directories under /var/lib/docker/overlay2 by hand.
• Storing application data in a container’s writable layer rather than a volume, so cleanup destroys it.
• Never running docker system df, and so deleting the wrong category entirely.
• Assuming inodes cannot be the problem.

Best practices

• Set log rotation in daemon.json on every host, on day one.
• Cap build cache with the builder GC settings rather than relying on manual pruning.
• Use named volumes for anything you care about, so cleanup and data are clearly separated.
• Use multi-stage builds and a real .dockerignore. Smaller images mean less to clean up.
• Automate only the safe operations, with an age filter, and log what the job removed.
• Alert on disk usage at 80% so cleanup is a scheduled task rather than an incident.
• On busy build hosts, give /var/lib/docker its own partition so a runaway cache cannot take the OS down with it.

Frequently asked questions

Is docker system prune safe to run?

Without flags, yes, on most systems. It removes stopped containers, unused networks, dangling images and build cache. Adding -a also removes tagged images with no container attached. Adding --volumes is the one that can lose data.

Will pruning delete my database?

Only if the database lives in a volume and you pass --volumes, or it lives in a container’s writable layer and you prune that container. If it is in a named volume attached to a running container, ordinary pruning will not touch it.

Why does docker system df not match du?

Two reasons. It does not count container logs at all, and it accounts for shared image layers once rather than per image. du -sh /var/lib/docker is the honest total.

How do I stop logs growing without limit?

Set max-size and max-file under log-opts in /etc/docker/daemon.json, restart the daemon, then recreate existing containers so they pick up the new configuration.

Can I move Docker’s storage to another disk?

Yes, with data-root in daemon.json. Stop Docker, copy /var/lib/docker across preserving ownership and extended attributes, point data-root at the new location, then start it again. Copy rather than move until you have confirmed it works.

Does pruning affect running containers?

No. Prune operations skip anything in use. The risk is always to things that are stopped, dangling or unattached — which is exactly why checking what is stopped before you prune is worth the thirty seconds.

Wrapping up

Docker disk usage is four categories that grow independently, plus a fifth — logs — that hides from Docker’s own accounting. Run docker system df, work down from build cache to volumes, and stop as soon as you have the space. The nuclear option exists, but reaching for it first is how people discover which of their volumes mattered.

Once the immediate problem is solved, spend ten minutes on log rotation and builder GC. Those two settings turn this from a recurring emergency into something you never think about again.

Need a hand with Docker in production?

I work with teams on container infrastructure — image sizes that got out of hand, build pipelines that cache badly, storage and logging defaults that were never set, and monitoring that catches a full disk before the alert does.

If you would rather not spend the evening on it, you can hire me on Upwork.

The post Cleaning Up Docker Disk Usage Safely appeared first on John Nessime.