You go back through the logs and there it is, four hundred lines up, in a step that reported success: a download that returned an HTML error page, got piped into tar, and produced an archive containing nothing useful. The shell reported the exit status of tar, not of curl. The build carried on. The pipeline was never lying on purpose. It just never asked the right question.

Two things go wrong with pipelines, and only one of them gets talked about. The famous one is slow feedback: thirty-four minutes of build and integration tests before a linter tells you a file is missing a trailing newline. The expensive one is a pipeline that passes when it should have stopped. A fail fast pipeline fixes the first problem. Making it explain itself fixes the second, and that is the half most teams skip.

This post covers both: how to order stages so real failures surface in seconds, how to stop the shell from hiding non-zero exits, how to design failure output that hands the next engineer the fix, and the troubleshooting steps for when a pipeline stubbornly refuses to fail at the right moment.

The failure that actually bites: a pipeline that lies

Start here, because everything else is optimisation on top. A slow pipeline wastes your afternoon. A silently passing pipeline ships a broken artifact and burns a day of debugging in production, usually while somebody is on a call with a customer.

The shell is where most of this happens. Bash gives you three defaults that are wrong for CI, and one line fixes all three.

#!/usr/bin/env bash
set -euo pipefail

# Without pipefail this line exits 0 even when curl fails,
# because the shell reports the status of the LAST command
# in the pipe, which is tar.
curl -sSL https://example.com/artifact.tar.gz | tar -xz

What each flag actually protects

• -e stops the script when a command exits non-zero. Without it, a failed database migration is followed by a successful echo, and the script exits 0 because the last command succeeded.
• -u turns an unset variable into an error. This is the one that saves you from rm -rf "$PREFIX/build" expanding to rm -rf "/build" when someone renames an environment variable in the pipeline settings and forgets the script.
• -o pipefail makes a pipeline return the first non-zero status instead of the last one. Every curl | tar, cat | jq, and docker logs | grep in your scripts is currently reporting only the exit code of the thing on the right.

The exit status is only half the story though. Plenty of tools return 0 while doing something useless, and curl is the classic offender.

# curl exits 0 here even when the server returns 500,
# and you end up untarring an HTML error page.
curl -sSL "$ARTIFACT_URL" | tar -xz

# --fail          makes curl exit non-zero on 4xx and 5xx
# --show-error    keeps the reason visible even with --silent
# --location      follows redirects instead of saving a 302 body
curl --fail --silent --show-error --location "$ARTIFACT_URL" | tar -xz

Now the download failing stops the build, and pipefail makes sure that failure propagates past the pipe. The first fixes the check, the second makes sure the check is heard.

Where set -e quietly gives up

This trips people up constantly. set -e is suspended whenever a command’s exit status is being consumed by something else. It is not a safety net, it is a default.

set -e

# -e does NOT stop the script here.
# The if statement consumes the exit status.
if grep -q "TODO" src/main.go; then
  echo "found"
fi

# -e does NOT stop the script here either.
# The left side of || is explicitly allowed to fail.
run_migrations || echo "migrations failed, continuing anyway"

# -e DOES stop the script here.
run_migrations

The same trap applies inside containers. On Debian-based images /bin/sh is dash, which has no pipefail option at all, so putting set -o pipefail inside a RUN line fails outright. Set the shell at the top of the Dockerfile instead. Hadolint flags this as DL4006 and it is worth wiring into your static stage.

# Set this before any RUN instruction that contains a pipe.
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl --fail --silent --show-error --location 
      https://example.com/tool.tar.gz | tar -xz -C /usr/local/bin

Order stages by cost, not by category

Most pipelines are laid out the way the team thinks about work: tests, then build, then security, then deploy. That grouping is comfortable and it is the wrong axis. Order by two things instead, cost and failure probability.

The rule I use: every stage should be cheaper than the one after it and more likely to fail than the one after it. If a check catches ten percent of your broken commits and takes fifteen seconds, it goes in front of the check that catches two percent and takes nine minutes. That is the whole design principle behind a fail fast pipeline.

1. Syntax and static analysis. shellcheck, hadolint, yamllint, terraform validate, tflint, formatter checks, type checks. Seconds, no dependencies, no network. This stage catches an embarrassing share of broken pushes.
2. Unit tests. No database, no containers, no network. If they need any of those, they are integration tests and belong further down.
3. Secrets and dependency scanning. Fast, and a leaked credential is worth stopping before you build an image around it.
4. Build. This is where you start spending real minutes and cache space.
5. Integration and end-to-end. Slow, flaky, expensive to run. Everything above should have already caught the boring failures.
6. Deploy and post-deploy verification.

A note on the trade-off, because this ordering costs something. Splitting the fast checks into their own stage adds a checkout and a container pull before your unit tests even start. On a busy shared runner that overhead is real. If your static stage takes twelve seconds of work and forty seconds of setup, merge it into the test job and run it as the first step instead. The principle is failure order, not job count.

The other thing that keeps the fast lane fast is caching, and it is the piece people get wrong in the opposite direction: they cache so aggressively that a stale dependency tree hides a genuine break. Cache the package manager’s download directory, not the resolved dependency tree, and key the cache on the lockfile hash.

Let the platform kill work you already know is doomed

Ordering stages is only useful if the platform actually stops when something breaks. Three settings do most of the work, and they exist under different names on every runner.

GitHub Actions

concurrency:
  # A new push to the same branch cancels the run that is
  # already in flight. Nobody needs results for a commit
  # that has been superseded.
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 5          # a hung job is worse than a failed one
    steps:
      - uses: actions/checkout@v4   # pin to a full commit SHA in production
      - run: shellcheck scripts/*.sh
      - run: hadolint Dockerfile

  test:
    needs: static               # will not start until static passes
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false          # see note below
      matrix:
        suite: [unit, contract]
    steps:
      - uses: actions/checkout@v4
      - run: make test-${{ matrix.suite }}

That fail-fast: false looks like it contradicts the whole post, so it is worth explaining. Matrix fail-fast defaults to true: one leg fails, the rest get cancelled. That is right when the legs are redundant and you only need to know that something broke. It is wrong when the legs tell you different things. If your matrix is three runtime versions and only one fails, cancelling the others hides whether this is a version-specific problem or a general one, and you will burn a rerun finding out. Fail fast on the pipeline, not on the diagnosis.

timeout-minutes is the setting people skip and regret. A job waiting on an interactive prompt, a dead package mirror, or a container that never becomes healthy will sit there consuming a runner until someone notices. Set a timeout on every job, generous enough that a slow day does not trip it and tight enough that a hang gets caught inside one coffee break.

GitLab CI

stages: [static, test, build, deploy]

default:
  interruptible: true     # allows auto-cancel when a newer pipeline starts
  retry:
    max: 2                # capped at 2 by GitLab, so 3 attempts total
    when:
      # Retry only on infrastructure problems.
      # Never retry script_failure: that is your code failing,
      # and retrying it just hides a flaky test.
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

lint:
  stage: static
  timeout: 5m
  script:
    - shellcheck scripts/*.sh

integration:
  stage: test
  needs: [lint]           # DAG: starts as soon as lint passes
  script:
    - make integration

The retry: when list is the important bit and it is the one people get lazy about. A bare retry: 2 retries on everything including a genuine test failure, which converts an intermittent bug into an invisible one. Scope retries to the failure classes that are genuinely the infrastructure’s fault. Jenkins has failFast true for parallel stages, Buildkite and CircleCI have their own equivalents, and the reasoning transfers cleanly.

A failure should hand you the answer, not the search

Here is the test I apply to a pipeline failure. Can someone who did not write the change, at an inconvenient hour, fix it from the failure output alone without opening the source? If not, the failure message is incomplete.

Four things need to be in there:

• What failed, named precisely. Not “build failed” but the command and its exit code.
• Why, in terms of the input rather than the internals. “lockfile is out of date with package.json”, not “npm ci exited 1”.
• Where, meaning the file, the line, or the config key that caused it.
• What to do next. One concrete action, even if it is just a runbook link.

In shell, a small helper plus an ERR trap gets you most of the way there.

#!/usr/bin/env bash
set -euo pipefail

fail() {
  echo "ERROR: $1" >&2
  echo "  fix: $2" >&2
  exit 1
}

# Catches anything you did not write an explicit check for,
# and tells you the line and the command that blew up.
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

[[ -f config/app.yaml ]] || fail 
  "config/app.yaml is missing" 
  "copy config/app.yaml.example and set DATABASE_URL"

[[ -n "${IMAGE_TAG:-}" ]] || fail 
  "IMAGE_TAG is empty" 
  "check the tag job output; it sets IMAGE_TAG from the git SHA"

Note ${IMAGE_TAG:-} rather than $IMAGE_TAG. Under set -u the bare form aborts with a shell error before your nice message ever prints, which is technically correct and practically useless.

On GitHub Actions you can go one step further and attach the error to the file and line, so it shows up as an annotation on the pull request instead of something to be dug out of a log.

# Renders as a red annotation on that exact line in the diff view.
echo "::error file=deploy/values.yaml,line=42::image tag is empty"

# Visible, but does not fail the job.
echo "::warning file=Dockerfile,line=7::apt cache not cleaned"

Two more habits worth building. Upload logs, test reports, and screenshots as artifacts on failure, conditionally, so the evidence survives the job that produced it. And emit test results as JUnit XML so the platform renders a list of failed test names instead of asking people to scroll. Both are cheap and both cut the time between “it broke” and “I know why” more than any clever retry logic will.

Exit codes are a contract, so use more than one

Most scripts have two states: 0 and “something went wrong”. That collapses distinctions the pipeline needs. Terraform gets this right and it is a good model to copy.

# -detailed-exitcode changes the meaning of the exit code:
#   0 = succeeded, no changes
#   1 = error
#   2 = succeeded, changes present
set +e
terraform plan -detailed-exitcode -out=tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift. Skipping apply." ;;
  2) echo "Changes detected. Holding for approval." ;;
  *) echo "Plan failed. See log above." >&2; exit "$code" ;;
esac

The set +e around it is deliberate: with -e active, exit code 2 would kill the script before you got to interpret it. This is one of the few places where turning the flag off is the correct move, and it is worth a comment in the file explaining why so nobody “fixes” it later.

Do the same in your own scripts. Reserve a code for “the thing you asked for is not possible” versus “I could not reach the API” versus “your input is wrong”. The pipeline can then retry one and refuse to retry the others, which is the difference between a retry that absorbs a runner hiccup and a retry that hides a race condition for six months.

Gate on what you care about, not what is easy to check

This is the deploy-stage version of the same lie. Plenty of commands confirm that a request was accepted, not that the thing you wanted happened.

# Returns 0 the moment the API server accepts the manifest,
# even if every pod goes into CrashLoopBackOff a second later.
kubectl apply -f k8s/

# This is the step that actually waits for the rollout,
# and fails the job if it does not converge.
kubectl rollout status deployment/api --timeout=180s

# And this proves the application answers a request,
# not just that a pod reports Ready.
curl --fail --silent --show-error https://api.example.com/healthz

Run that last check against the public hostname, through your CDN, rather than against the pod directly. If you front the app with Cloudflare or similar, a smoke test that skips the edge will happily pass while real users hit a cached error page or a misrouted origin. Same reasoning applies to the origin host itself: if the app runs on a VPS from a provider like InterServer behind a reverse proxy, verify through the proxy, because that is the path traffic takes.

Then track how long each stage takes over time. Pipeline duration and failure rate per stage are perfectly ordinary metrics to ship into Prometheus and graph in Grafana, or into whatever platform you already use for application monitoring. When the fast lane creeps from forty seconds to four minutes, that is a slow regression nobody will notice from a single run.

Troubleshooting

The job passed but the artifact is wrong

Look for pipes first. Grep the scripts for | and check whether pipefail is set in that shell. Then check for tools that return 0 on failure: curl without --fail, docker run where the entrypoint swallows errors, anything wrapped in || true that somebody added during an incident and never removed.

set -e is set but the script keeps going

The failing command is almost certainly inside a condition, on the left of || or &&, or in a function that is itself being called from an if. Add set -x temporarily and read the trace to see exactly where control flow continued. Then use an explicit check with your fail helper rather than relying on -e at that point.

Works locally, fails in CI (or the reverse)

Usually a shell difference. Your terminal runs bash or zsh, the runner may execute the script with sh. Add an explicit shebang, and on GitHub Actions set the shell on the step or at the job level rather than assuming. Inside containers, remember dash has no pipefail and no [[ ]].

The job hangs until the platform kills it

Something is waiting on stdin. Package managers prompting for confirmation, ssh asking about a host key, a database client waiting for a password. Add the non-interactive flags, set CI=true where tools respect it, and put a timeout on the job so the next occurrence fails in ten minutes instead of sixty.

Fails on the first run, passes on rerun

Resist the retry. Log the failure with enough context to identify it later, quarantine the test into a non-blocking job if it is genuinely blocking the team, and give it an owner and a date. A blanket retry turns a flaky test into a permanent tax on everyone’s confidence in the pipeline.

The failure output is a wall of noise

Suppress progress bars and spinners in CI, they render as thousands of unreadable lines. Then print a short summary block at the very end of the job, after the noise, listing what failed and what to do. People read the bottom of a log first.

Common mistakes

• Writing set -e without -o pipefail and assuming the script is covered.
• Using || true to silence a step that fails intermittently, instead of finding out why.
• Running the slowest, flakiest checks before the ones that take fifteen seconds.
• Retrying on every failure class rather than only infrastructure faults.
• No job timeouts, so a hang consumes a runner until someone notices.
• Treating kubectl apply or a deploy API call as proof the deploy worked.
• Failure messages that name the internal function that threw, and nothing the reader can act on.
• Caching so aggressively that a stale dependency tree hides a real break.
• Discarding logs and test reports when the job fails, which is exactly when you need them.
• Turning off matrix fail-fast everywhere by reflex, so every broken commit costs the full matrix.

Best practices

• Put set -euo pipefail at the top of every shell script the pipeline runs, and set the Dockerfile SHELL before any piped RUN.
• Order stages so each one is cheaper and more likely to fail than the next.
• Give every job a timeout, and cancel superseded runs on the same branch.
• Use explicit checks with actionable messages instead of relying on -e to catch everything.
• Distinguish exit codes so the pipeline can react differently to different failures.
• Verify deploys with a rollout status check plus a real HTTP request through the public path.
• Upload artifacts and structured test reports on failure.
• Pin action and image references to immutable digests so a green pipeline stays reproducible.
• Track stage duration and failure rate over time, the same way you track application metrics.
• Fix the failure message the first time somebody has to ask what it means.

FAQ

What does “fail fast” actually mean in a CI/CD pipeline?

Two things. First, the pipeline stops as soon as it knows the result will be a failure, rather than finishing work whose outcome no longer matters. Second, the cheapest checks that are most likely to catch a problem run first, so the average failure surfaces in seconds. The second half is a design choice about ordering, not a platform setting.

Is set -e enough on its own?

No. It is suspended inside conditions, on the left side of && and ||, and after !. It also does nothing about pipes, which is why -o pipefail matters, or about commands that return 0 while failing, which is why explicit checks matter. Use it as a baseline, not as your only line of defence.

Should I disable matrix fail-fast?

Only when the legs give you different diagnostic information. If the matrix is runtime versions or operating systems and you need to know whether the break is universal or specific, let all legs run. If the legs are interchangeable shards of the same test suite, leave fail-fast on and save the compute.

How do I stop a pipeline from passing when a step really failed?

Work through three things in order. Set pipefail so pipes propagate failure. Add the flags that make tools return non-zero on failure, such as curl --fail. Then replace implicit trust with explicit assertions: after a deploy, check that the rollout converged and that the service answers a request. Most silent passes come from one of those three gaps.

Are automatic retries a good idea?

Only for failures that are the infrastructure’s fault: runner crashes, image pull errors, API timeouts from the CI platform itself. Never retry a script failure by default. And make retries visible, because a job that quietly succeeds on the third attempt is telling you something important that nobody sees.

How long should a fast feedback stage take?

Short enough that an engineer waits for it rather than switching context. Somewhere under a couple of minutes for the combined lint, format, and unit stage is a reasonable target for most codebases. Past that, people push and walk away, and you lose the benefit of fast feedback even though the pipeline is technically quick.

Does this apply to self-hosted runners too?

All of it, and timeouts matter more. On hosted runners a stuck job eventually hits the platform’s own ceiling. On your own machine it holds a slot until you intervene, so a hung job blocks the queue for everyone. Set explicit timeouts and make sure the runner cleans up its workspace between jobs.

The one thing to take away

Speed is the part everyone optimises and honesty is the part that actually costs money. A fail fast pipeline that finishes in ninety seconds is worth very little if one of those seconds contains a swallowed exit code. Fix the lying first: pipefail, tools that return non-zero when they fail, and assertions that check the outcome rather than the request. Then reorder for speed.

And judge every failure by one question. Could someone else fix this from the output alone, without reading the source? If the answer is no, the pipeline has not finished its job, no matter how quickly it went red.

Need help fixing a pipeline that lies to you?

If any of this sounds like your CI, this is the kind of work I take on. Typical engagements look like:

• Auditing an existing GitHub Actions, GitLab CI, or Jenkins pipeline for silently swallowed failures and false-green stages.
• Restructuring stage order and caching so the common failure surfaces in seconds instead of after the build.
• Hardening deploy verification: rollout gates, smoke tests through the real public path, and automatic rollback on a failed health check.
• Rewriting pipeline shell scripts with proper error handling, exit code contracts, and failure messages an on-call engineer can act on.
• Wiring pipeline duration and failure metrics into Prometheus and Grafana so regressions get caught early.

Send me the workflow file and a link to a failing run, and I will tell you what I would change first.

The post Green Check, Broken Build: How to Build a Fail Fast Pipeline That Explains Why appeared first on John Nessime.

Docker disk usage grows quietly because every part of it accumulates independently — images, stopped containers, volumes, build cache, and container logs, which are the one nobody expects and which no Docker command reports on. Cleaning it up is easy. Cleaning it up without destroying something takes about five more minutes.

This is the order I work through it: measure first, reclaim from safest to most destructive, then fix the thing that caused the growth so you are not back here next month.

Measure before you delete

Every cleanup starts here, and skipping it is how people delete the wrong thing:

docker system df

That gives you four rows — Images, Containers, Local Volumes, Build Cache — with a total size and a reclaimable figure for each. The reclaimable column is the one that matters. If build cache is eleven gigabytes and all of it is reclaimable, you have found your win and you never need to go near volumes.

For the itemised version:

docker system df -v

Now you can see individual images with their sizes, which containers are using what, and which volumes have no container attached. Read this before running anything destructive.

Two more views worth having open:

# What is actually running, and what is just sitting there stopped
docker ps -a --format 'table {{.Names}}t{{.Image}}t{{.Status}}t{{.Size}}'

# Images by age, oldest first
docker images --format 'table {{.Repository}}:{{.Tag}}t{{.Size}}t{{.CreatedSince}}'

Reclaim in order, safest first

These are ordered deliberately. Work down the list and stop as soon as you have enough space. Most of the time you never reach step four.

1. Stopped containers

# See what would go
docker ps -a --filter status=exited --filter status=created

docker container prune

This removes stopped containers and their writable layers. Anything running is untouched, and named volumes attached to those containers survive — only the container’s own layer goes.

The one caveat: if a stopped container held state in its writable layer rather than in a volume, that state is gone. That is a design problem rather than a cleanup problem, but it is worth a glance before you confirm.

2. Build cache

On any machine that builds images regularly, this is usually the biggest single win and it is completely safe. Build cache is derived data. Deleting it costs you a slower next build and nothing else.

# Everything older than a week
docker builder prune --filter "until=168h"

# Everything, no exceptions
docker builder prune -a

3. Dangling images

# Look first
docker images -f dangling=true

docker image prune

Without -a, this removes only dangling images — untagged layers left behind when a rebuild moved a tag to a new image. They are almost always genuine garbage.

4. Unused images — read this one carefully

docker image prune -a

Adding -a changes the meaning completely. It removes every image that does not have at least one container associated with it — running or stopped. So an image you pulled for a job that runs weekly, with no container currently existing, is gone. It will pull again, which is fine if you have bandwidth and the registry is reachable, and much less fine at 3am on a machine with a slow link.

Safer in most cases is an age filter, which keeps anything recent:

docker image prune -a --filter "until=336h"   # older than 14 days

5. Volumes — the one that loses data

Volumes are where databases live. Treat this step as a data operation, not a cleanup.

Before removing anything, look inside the candidates. This mounts each dangling volume read-only into a throwaway container so you can see what is actually in it:

docker volume ls -qf dangling=true | while read -r v; do
  echo "=== $v"
  docker run --rm -v "$v":/vol:ro alpine sh -c 'du -sh /vol; ls -la /vol' 2>/dev/null | head -12
done

If you see anything resembling a data directory, back it up before it goes:

docker run --rm 
  -v my-volume:/vol:ro 
  -v "$PWD":/backup 
  alpine tar czf /backup/my-volume.tar.gz -C /vol .

Only then:

docker volume prune

On recent Docker versions this prunes only anonymous volumes by default, with --all needed to include named ones — a genuinely good safety change. Older versions removed named volumes too. Check which behaviour you have before trusting it:

docker version --format '{{.Server.Version}}'
docker volume prune --help

The gigabytes Docker never tells you about

Here is the part that catches experienced people. Container logs do not appear in docker system df at all, and no prune command touches them. A chatty application with the default logging driver will happily write tens of gigabytes and nothing in Docker’s own tooling will mention it.

sudo du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -10

To map a log file back to the container that owns it:

docker ps -aq | xargs docker inspect --format '{{.Name}} {{.LogPath}}'

Truncate, do not delete

This matters. Deleting a log file that a running container still has open removes the directory entry but not the data — the kernel keeps the blocks allocated until the process closes the descriptor, so df shows no improvement and you are left confused. Truncate instead:

sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Then stop it happening again

Set a global default in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

sudo systemctl restart docker

One thing people get wrong here: this applies to newly created containers only. Anything already running keeps the logging config it was created with. You have to recreate those containers for the limit to take effect — restarting them is not enough.

While you are in that file, cap the build cache too:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  }
}

Automating it without automating a disaster

A weekly cron is worth having, provided it only ever does the safe operations. The rule is simple: never put --volumes in anything unattended.

# /etc/cron.d/docker-prune
# Weekly, Sunday 03:00. Containers, networks, dangling images and build
# cache older than seven days. No volumes, ever.
0 3 * * 0 root /usr/bin/docker system prune -f --filter "until=168h" >> /var/log/docker-prune.log 2>&1

Note what docker system prune covers without flags: stopped containers, unused networks, dangling images and build cache. It does not touch volumes unless you explicitly pass --volumes, and it does not remove tagged images unless you pass -a. Those two flags are the whole difference between a routine job and an incident.

If you are tempted to add --volumes to a cron job so you do not have to think about it again, you have swapped a disk space problem for a data loss problem on a timer.

Troubleshooting

Pruned everything, df has not moved

Deleted files still held open by a running process. Find them:

sudo lsof +L1 | head -20

Restarting the process that holds the descriptor releases the blocks. If it is the Docker daemon itself, a systemctl restart docker does it, at the cost of bouncing every container without a restart policy.

“No space left on device” but df shows free space

You are out of inodes rather than bytes. Docker’s layered storage creates enormous numbers of small files:

df -i

The fix is the same — prune images and build cache, which is where the file count lives.

overlay2 is huge and I have hardly any images

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2 2>/dev/null

Usually build cache or container writable layers rather than images. Do not delete anything under overlay2 by hand — the directory names are referenced in Docker’s metadata, and removing them directly corrupts the daemon’s view of the world. Always go through docker commands.

Running low and need space right now

Fastest safe sequence, in order:

docker builder prune -a -f
docker container prune -f
docker image prune -f
sudo truncate -s 0 /var/lib/docker/containers/*/*-json.log

Four commands, no volumes touched, no tagged images removed. That resolves the large majority of Docker disk usage emergencies.

Common mistakes

• Reaching for docker system prune -a --volumes first because it is the top search result.
• Putting --volumes in a cron job.
• Deleting log files instead of truncating them, then wondering why df did not change.
• Setting log limits in daemon.json and assuming existing containers pick them up. They do not.
• Removing directories under /var/lib/docker/overlay2 by hand.
• Storing application data in a container’s writable layer rather than a volume, so cleanup destroys it.
• Never running docker system df, and so deleting the wrong category entirely.
• Assuming inodes cannot be the problem.

Best practices

• Set log rotation in daemon.json on every host, on day one.
• Cap build cache with the builder GC settings rather than relying on manual pruning.
• Use named volumes for anything you care about, so cleanup and data are clearly separated.
• Use multi-stage builds and a real .dockerignore. Smaller images mean less to clean up.
• Automate only the safe operations, with an age filter, and log what the job removed.
• Alert on disk usage at 80% so cleanup is a scheduled task rather than an incident.
• On busy build hosts, give /var/lib/docker its own partition so a runaway cache cannot take the OS down with it.

Frequently asked questions

Is docker system prune safe to run?

Without flags, yes, on most systems. It removes stopped containers, unused networks, dangling images and build cache. Adding -a also removes tagged images with no container attached. Adding --volumes is the one that can lose data.

Will pruning delete my database?

Only if the database lives in a volume and you pass --volumes, or it lives in a container’s writable layer and you prune that container. If it is in a named volume attached to a running container, ordinary pruning will not touch it.

Why does docker system df not match du?

Two reasons. It does not count container logs at all, and it accounts for shared image layers once rather than per image. du -sh /var/lib/docker is the honest total.

How do I stop logs growing without limit?

Set max-size and max-file under log-opts in /etc/docker/daemon.json, restart the daemon, then recreate existing containers so they pick up the new configuration.

Can I move Docker’s storage to another disk?

Yes, with data-root in daemon.json. Stop Docker, copy /var/lib/docker across preserving ownership and extended attributes, point data-root at the new location, then start it again. Copy rather than move until you have confirmed it works.

Does pruning affect running containers?

No. Prune operations skip anything in use. The risk is always to things that are stopped, dangling or unattached — which is exactly why checking what is stopped before you prune is worth the thirty seconds.

Wrapping up

Docker disk usage is four categories that grow independently, plus a fifth — logs — that hides from Docker’s own accounting. Run docker system df, work down from build cache to volumes, and stop as soon as you have the space. The nuclear option exists, but reaching for it first is how people discover which of their volumes mattered.

Once the immediate problem is solved, spend ten minutes on log rotation and builder GC. Those two settings turn this from a recurring emergency into something you never think about again.

Need a hand with Docker in production?

I work with teams on container infrastructure — image sizes that got out of hand, build pipelines that cache badly, storage and logging defaults that were never set, and monitoring that catches a full disk before the alert does.

If you would rather not spend the evening on it, you can hire me on Upwork.

The post Cleaning Up Docker Disk Usage Safely appeared first on John Nessime.

The Pulumi vs Terraform question got harder in the last two years, not easier. The licence changed, the community forked, IBM bought HashiCorp, HashiCorp killed its own general-purpose-language option, and Pulumi started speaking HCL. A lot of the comparison articles you will find were written before any of that happened.

This is an attempt at an honest read of where things actually stand, what the real trade-off is once you strip away the marketing, and how to pick a lane you will not regret in eighteen months.

What actually changed

A short timeline, because the current landscape only makes sense against it.

• August 2023 — HashiCorp relicensed Terraform from MPL 2.0 to the Business Source License. Source-available, free for provisioning your own infrastructure, but you cannot build a competing commercial product on it.
• January 2024 — OpenTofu shipped 1.6.0, a fork of the last MPL-licensed code, governed by the Linux Foundation.
• February 2025 — IBM completed its acquisition of HashiCorp, valued at around $6.4 billion.
• December 2025 — HashiCorp deprecated CDK for Terraform and archived the repository. The official way to write Terraform in a general-purpose language no longer exists.
• Also December 2025 — Pulumi announced native HCL support and the ability to work with Terraform and OpenTofu state.

Those last two matter more than the licence drama, and they point in opposite directions. HashiCorp closed the door on writing Terraform in a real language. Pulumi opened the door to writing Pulumi in HCL. Whatever else you conclude, “HCL versus real languages” is no longer a clean line between two products.

The real axis: a configuration language or a programming language

Strip out the licensing and the corporate ownership and one genuine difference remains. HCL is a purpose-built configuration language that describes desired state. Pulumi runs an actual program that constructs a resource graph, and then applies it.

Here is the same trivial thing in both. Terraform:

resource "aws_s3_bucket" "logs" {
  for_each = toset(var.environments)
  bucket   = "acme-logs-${each.key}"

  tags = {
    Environment = each.key
    ManagedBy   = "terraform"
  }
}

And Pulumi, in TypeScript:

import * as aws from "@pulumi/aws";

const environments = ["dev", "staging", "prod"];

const buckets = environments.map(env =>
  new aws.s3.Bucket(`logs-${env}`, {
    bucket: `acme-logs-${env}`,
    tags: {
      Environment: env,
      ManagedBy: "pulumi",
    },
  })
);

For something this small the difference is aesthetic. It stops being aesthetic around the third nested conditional.

What you actually gain with a general-purpose language

• Real abstractions. Classes, functions, inheritance. A Pulumi component that encapsulates “our standard service” is just a class, and it composes the way code composes.
• Real testing. Jest, pytest, Go’s testing package. You can unit test infrastructure logic with the same tools and the same CI as the rest of your codebase.
• The package ecosystem. Need to parse a CSV of subnets, call an internal API, or do something genuinely awkward? It is an npm install rather than an external data source and a shell script.
• IDE support that works. Type checking, refactoring, jump-to-definition. HCL tooling has improved a lot but it is not the same thing.

What you lose, which comparison posts skip

This is the part worth reading twice, because it is where the regret comes from.

• Constraint is a feature. HCL is limited on purpose. When a language cannot express clever things, nobody writes clever things. Give a team a real language and someone will eventually build a metaprogramming layer that generates resource names from a database lookup, and reviewing that pull request will be nobody’s idea of a good afternoon.
• Readability across skill levels. A network engineer who has never written TypeScript can usually read HCL and tell you what it provisions. The reverse is not true. If your infrastructure is touched by people who are not primarily programmers, that matters enormously.
• The hiring pool. There are far more engineers with Terraform on their CV than Pulumi. That is not an argument about quality, it is an argument about how long it takes to backfill a role.
• A dependency on a runtime. Pulumi programs need Node, Python, or Go present and correct in every environment that runs them, including CI. It is one more thing that can drift.
• Asynchrony leaks into the model. Resource outputs are not plain values. They are futures, and you have to handle them as such. Every Pulumi team hits this on day two.

# This does not do what a newcomer expects. bucket.id is an Output,
# not a string, so you get an object reference in the name.
# bucket_name = f"logs-for-{bucket.id}"

# Correct: work inside apply, where the value is resolved.
bucket_name = bucket.id.apply(lambda bid: f"logs-for-{bid}")

None of these are dealbreakers. They are just the bill that arrives later, and it is worth reading before you sign.

State, providers, and the things that actually break

State management

Both tools keep a state file mapping your code to real resources, and both will ruin your afternoon if that file gets corrupted or out of sync.

Terraform and OpenTofu use pluggable backends: S3 with DynamoDB locking, Azure Blob, GCS, Consul, or HCP Terraform. OpenTofu added client-side state encryption, which Terraform does not have natively and which is a genuine advantage if your compliance people care about what is sitting in that bucket.

Pulumi defaults to Pulumi Cloud, which is free for individuals and paid for teams. It also supports self-managed backends, and you should know this because the default sometimes gets mistaken for a requirement:

# Self-managed state in S3, no Pulumi Cloud account involved
pulumi login s3://my-pulumi-state-bucket

# Or entirely local, useful for experiments
pulumi login --local

Pulumi encrypts secrets in state by default. With Terraform, values marked sensitive are redacted in output but still sit in plaintext in the state file, which surprises people who assumed otherwise.

The provider ecosystem, and a myth worth killing

You will read that Pulumi has a much smaller provider ecosystem. That was true once and is mostly no longer the point, because Pulumi bridges Terraform providers. If a Terraform provider exists, you can almost certainly use it from Pulumi, either pre-built in the Pulumi Registry or generated on demand.

Two caveats that are real:

• Bridged providers can lag upstream releases. If you need a resource type that shipped last week, check before committing.
• When something breaks deep in a bridged provider, you are debugging across a translation layer. That is a worse afternoon than debugging the provider directly.

Pulumi also ships native providers for Kubernetes and Azure, generated from the platform APIs, which give same-day coverage of new resources rather than waiting on a bridge.

Policy and governance

Terraform’s first-class answer is Sentinel, which is tied to HCP Terraform. OpenTofu users generally reach for OPA or Conftest. Pulumi has CrossGuard, where policies are written in the same languages as everything else.

If you are in a regulated environment and your auditors already know what Sentinel is, that is a heavier thumb on the scale than any language preference.

Licensing, since it is what forced the question

• Terraform — BSL 1.1. Source-available. Fine for provisioning your own infrastructure. Not fine for building a product that competes with HashiCorp. Each release converts to an open licence four years after publication, but only that release.
• OpenTofu — MPL 2.0, OSI-approved, Linux Foundation governance. No single vendor can relicense it.
• Pulumi — Apache 2.0 for the engine and CLI. Permissive, with a patent grant. Pulumi Corp monetises through Pulumi Cloud rather than through the licence.

For the overwhelming majority of teams, the BSL changes nothing about daily work. You are provisioning your own infrastructure and that is explicitly permitted. The concern is not today’s terms, it is that a single vendor demonstrated it can change them, and that vendor is now part of IBM.

If your objection to Terraform is licensing rather than language, the answer is OpenTofu, not Pulumi. Switching paradigms to solve a legal problem is an expensive way to do it.

Choosing a lane

Three coherent positions. Pick the one that matches your constraints rather than the one that sounds most modern.

Lane one: OpenTofu

The default for most teams already running Terraform who are uneasy about the licence. Migration is genuinely close to free: same HCL, same providers, same state format, same commands.

# Back up state first. Always.
terraform state pull > backup-$(date +%F).tfstate

# Then initialise with tofu against the same backend
tofu init
tofu plan

A clean plan with no proposed changes means you are done. That is the whole migration for most codebases.

Choose this if: you have existing Terraform, HCL is working for you, and you want open governance without retraining anyone.

Lane two: Terraform with HCP

Still the biggest ecosystem, the most documentation, the most Stack Overflow answers, the most people who already know it. Stacks and ephemeral resources are Terraform-only. Sentinel is Terraform-only. If you are deep in HCP Terraform and it works, the licence is an abstract concern rather than a practical one.

Choose this if: you are in a regulated industry, you need vendor support with a contract behind it, or your organisation is standardising on IBM and Red Hat tooling anyway.

Lane three: Pulumi

The strongest case for Pulumi is not “real languages are nicer”. It is when your infrastructure genuinely needs logic that HCL fights you on: dynamic topologies, infrastructure derived from application code, platform teams building abstractions for other teams to consume. If you are writing a self-service platform rather than a set of environments, a real language stops being a preference and starts being the right tool.

Choose this if: your team is primarily software engineers, your infrastructure has real branching logic, or you are building a platform product rather than provisioning a fixed estate.

Do not choose it because writing loops in HCL annoyed you once.

What migrating actually looks like

Terraform to OpenTofu is a swap. Terraform to Pulumi is a refactor, and anyone telling you otherwise is selling something.

Pulumi does provide real tooling for it:

# Convert HCL into a Pulumi program in your chosen language
pulumi convert --from terraform --language typescript

# Adopt the resources described by an existing state file
pulumi import --from terraform ./terraform.tfstate

Pulumi can also consume Terraform modules directly, so existing module investment is not automatically wasted. That is the single most useful fact for anyone considering a gradual move.

A sane sequence:

1. Pick one small, low-risk stack. Not the production VPC.
2. Convert it, import the state, and run a preview. The only acceptable result is no changes.
3. Refactor the generated code. Converted output is mechanically correct and stylistically awful, which is fine, but do not leave it that way.
4. Run both tools side by side for a while. Coexistence is supported; a big-bang cutover is not necessary.
5. Only then decide whether to move anything else.

If step two produces a diff you cannot explain, stop. An unexplained diff during an IaC migration is how resources get destroyed and recreated.

Common mistakes

• Switching tools to solve a licensing concern when OpenTofu solves it for free.
• Picking the tool the loudest engineer prefers rather than the one the whole team can maintain.
• Adopting CDKTF in 2026. It was archived in December 2025 and will not receive fixes.
• Treating a converted program as finished code rather than a starting point.
• Assuming Terraform state is safe because values are marked sensitive. Encrypt the backend.
• Migrating production first because it is the stack that matters most. It is the stack that matters most.
• Running two tools against the same resources without a clear ownership boundary.

Best practices whichever lane you pick

• Remote state with locking and encryption, from the first commit. Never local state for anything shared.
• Small stacks with clear boundaries. One enormous state file is the most common self-inflicted wound in this field.
• Plan or preview output in every pull request, and require a human to read it.
• Pin provider versions. Floating versions turn an unrelated deploy into an incident.
• Policy as code before you need it, not after the audit.
• Document the decision. Six months from now nobody will remember why, and the reasoning is worth more than the choice.

Frequently asked questions

Is Pulumi better than Terraform?

Neither is better in the abstract. Pulumi suits teams of software engineers building infrastructure with real logic in it. Terraform and OpenTofu suit teams provisioning a defined estate where constraint and readability matter more than expressiveness.

Can I use Terraform providers with Pulumi?

Yes. Pulumi bridges Terraform providers, either pre-built in its registry or generated on demand, and can consume Terraform modules directly. Bridged providers can lag upstream releases, so verify coverage for anything recent.

Should I move from Terraform to OpenTofu?

If licensing is your concern, it is close to free: same HCL, same providers, same state. Back up state, run tofu init and tofu plan, and confirm no changes. If you rely on Stacks, Sentinel, or HCP Terraform, staying put is reasonable.

What happened to CDKTF?

HashiCorp deprecated it on 10 December 2025 and archived the repository. It gets no further fixes or compatibility updates. If you want general-purpose languages for infrastructure now, Pulumi is the maintained option.

Does the BSL licence stop me using Terraform commercially?

Not for provisioning your own infrastructure, which is what almost everyone does. It restricts building a competing commercial IaC product on top of it. Check with your own legal team rather than a blog post if you are anywhere near that line.

Can Pulumi and Terraform coexist?

Yes, and for any migration of size that is the sensible approach. Pulumi can reference Terraform state and consume Terraform modules. The rule is that each resource has exactly one owner.

Wrapping up

The Pulumi vs Terraform decision is less about the tools than about who maintains your infrastructure and what it has to do. If your estate is a known set of environments maintained by people with mixed backgrounds, a constrained configuration language is a feature and OpenTofu gives you that without licensing anxiety. If you are building a platform, with branching logic and abstractions other teams consume, a real language earns its complexity and Pulumi is the maintained way to get one.

What has genuinely changed is that the lanes are no longer walled off. Pulumi speaks HCL and reads Terraform state. OpenTofu is a drop-in for Terraform. Converting between them is tooling rather than a rewrite. That makes the decision less permanent than it feels, which is a good reason to pick deliberately and a bad reason to agonise.

Pick the lane your team can still maintain when the person who chose it has moved on. Then write down why.

Need help choosing or migrating?

I work with teams on infrastructure as code decisions and the migrations that follow — Terraform to OpenTofu, Terraform to Pulumi, splitting up state files that grew too large, and wiring the whole thing into CI with policy checks that actually run.

If you would rather talk it through with someone who has untangled a few of these, you can hire me on Upwork.

The post Pulumi vs Terraform: Choosing a Lane appeared first on John Nessime.