The crontab entry is still there. The script still works when you run it by hand. Nothing in the monitoring went red, because nothing was watching whether the job ran at all. It was watching whether the server was up.

That is the real question behind cron vs systemd timers, and it has almost nothing to do with syntax. Both tools can run a command at 2 a.m. The difference is what happens on the night the command doesn’t run, and how long it takes anyone to find out.

This post compares the two by failure family rather than by feature list: missed runs, overlapping runs, environment surprises, output that goes nowhere, and jobs that eat the machine. Then a decision procedure, the same job written both ways, and how to debug each one when it goes quiet.

The failure that actually decides it

Scheduled jobs fail silently far more often than they fail loudly. A job that crashes with a stack trace gets fixed the same week. A job that stopped firing six weeks ago gets found during an incident.

Cron’s default answer to “what happened?” is to mail the job’s output to the owning user. On a server with no local mail transfer agent configured, which is most servers built in the last decade, that output is generated and then discarded. The daemon logs that it started the command. It does not log what the command printed, and in most implementations it does not log the exit status either.

So you get a log line saying cron ran your backup, every night, forever, including on the nights the backup wrote nothing. That line is the trap. It looks like evidence.

systemd timers hand this to journald instead. Standard output and standard error from the triggered service land in the journal, tagged with the unit name, alongside the exit status and the runtime. That is the single biggest practical difference between the two, and it is the reason most of my new scheduled work goes into timers.

Neither one alerts you. Both will happily not run for a month while you sleep well. Observability is not monitoring, and I’ll come back to that.

Where cron still wins

Cron is one line in one file. There is no unit to reload, no second file to keep in sync, no install section to get wrong. crontab -e, five fields, a command, done. For a job that trims a log directory on a box you own, that economy is not nothing.

It also runs where systemd does not. BSD hosts, Alpine images, containers, appliance firmware, embedded boxes, older enterprise systems still on a legacy init. If your configuration management has to cover a fleet that is not uniformly systemd, cron is the common denominator and it is not close.

Per-user scheduling is genuinely simpler too. A user runs crontab -e and owns their own jobs, with no root involvement and no unit files in /etc. On shared hosting and on control panels like DirectAdmin or cPanel, this is the only model available, and the panel writes the crontab for you.

Where cron loses

• No dependency ordering. Cron cannot wait for the network to be up or the database to be ready. It fires at the wall clock time and hopes.
• No overlap protection. If the 2 a.m. run is still going at 3 a.m., the 3 a.m. run starts anyway, on top of it.
• No resource containment. A runaway job competes with production for CPU, memory and disk with nothing standing between them.
• Minute resolution, and no catch-up if the machine was off.
• A syntax with sharp edges that fail quietly, which is the next section.

Where systemd timers win

A timer separates when from what. The .timer unit holds the schedule. The .service unit holds the job. That split feels like bureaucracy until the first time you need to run the job right now, out of band, without touching the schedule. Then it is just systemctl start myjob.service, and the run is logged exactly like a scheduled one.

Everything systemd knows how to do to a service, it will do to a scheduled job. Ordering after other units. Restart-on-failure. Runtime caps. Memory and CPU limits through cgroups. A dedicated user. A private /tmp. A failure handler unit that fires when the job exits non-zero.

You also get a schedule you can interrogate before it bites you. systemctl list-timers prints every timer on the box with its next and last elapse. systemd-analyze calendar takes an expression and tells you exactly when it would fire, which is the closest thing either scheduler has to a unit test.

Where systemd timers lose

• Two files, a daemon-reload and an enable for every job. That is real friction when you have forty small jobs.
• The calendar syntax is more expressive than cron’s and harder to eyeball. You will reach for systemd-analyze calendar more often than you expect.
• Portability is gone. These units run on systemd hosts and nowhere else.
• User timers need lingering enabled, or they stop when the user logs out and never start at boot. This surprises people once, memorably.
• The journal is not a log file. If your log shipping only tails /var/log, timer output is invisible to it until you point the shipper at journald.

Cron vs systemd timers, by failure family

The machine was off at 2 a.m.

Cron has no memory. If the box was down, rebooting, or suspended when the schedule came round, that run is gone. Nothing catches it up and nothing records that it was skipped.

The traditional fix is anacron, which tracks the last successful run per job in a timestamp file and runs the job shortly after boot if the interval has lapsed. It is why the scripts in /etc/cron.daily still run on a laptop that is closed every night. The catch is that anacron works in whole days and only covers the cron.daily, cron.weekly and cron.monthly directories by default. It does not help your custom job at 02:00.

systemd folds that behaviour into a single directive:

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

With Persistent=true, systemd writes a timestamp under /var/lib/systemd/timers/ each time the timer fires. On boot it compares that stamp against the schedule, and if a run was missed it triggers immediately. This matters more than it sounds on any VPS you reboot for kernel updates, and a great deal on hosts at providers like Contabo or InterServer where you take maintenance windows on their timetable rather than yours.

One caveat worth knowing: “immediately” means immediately. If you have twelve persistent timers and the box has been off for a week, all twelve fire within seconds of boot, on a machine that is still starting services. Pair Persistent=true with RandomizedDelaySec= so the catch-up is spread out.

Two copies of the same job

This is the one that corrupts data. A sync job normally takes four minutes and runs every five. One night the remote end is slow, the run takes eleven minutes, and cron starts two more copies while the first is still writing.

Cron will not stop this. You have to do it yourself, and the standard tool is flock from util-linux:

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

-n means fail immediately rather than queue behind the running copy. Without it you build a backlog of processes all waiting for the same lock, which turns an overlap problem into a fork bomb with a slow fuse.

systemd gets this for free from its own model. A unit is either active or it isn’t. If sync.service is still running when the timer elapses, there is no second copy to start, because starting an already-active unit is a no-op. You do not have to remember anything.

One default is worth knowing. A calendar timer schedules its next elapse from the previous trigger time, so if a run overran, that next elapse is already in the past and the service fires again the moment it finishes. Newer systemd releases add a DeferReactivation= boolean that schedules from when the service went inactive instead, so an overrunning job waits for the next real slot. Check your distribution’s systemd version before relying on it.

The environment your job actually gets

“It works when I run it manually” is the single most common scheduled-job bug, and both schedulers cause it, for the same reason: neither one gives your job an interactive login shell.

Cron runs the command with a short, hardcoded environment. Your shell profile is not sourced. Your PATH is much shorter than the one you tested with, which is why a script that calls docker, wp, aws or a version-manager shim works from your terminal and fails from cron. The working directory is the user’s home, not wherever you happened to be standing.

Cron adds one hazard that is entirely its own. Inside a crontab, the percent sign is a metacharacter: it becomes a newline, and everything after the first one is fed to the command as standard input. So this looks fine and is broken:

# Truncated at the first %. The redirect never happens.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

# Escaped. This is what you meant.
0 2 * * * /usr/bin/mysqldump app > /backups/app-$(date +%F).sql

There is no warning. The line parses, cron runs the truncated fragment, and the log says the job started. Anything with a date + format string in a crontab deserves a second look.

systemd services get a minimal environment too, so the class of bug is identical. What differs is that the fix is declarative and lives with the unit rather than being hidden inside a wrapper script:

[Service]
Type=oneshot
User=deploy
WorkingDirectory=/srv/app
Environment=PATH=/usr/local/bin:/usr/bin:/bin
EnvironmentFile=/etc/myapp/backup.env
ExecStart=/usr/local/bin/backup.sh

The honest summary: absolute paths for every binary solve most of this on either scheduler. The rest is solved by never putting logic in the schedule. Put it in a script and schedule the script.

Everything firing at once

Write 0 0 * * * on fifty hosts and fifty hosts hit your backup target, your package mirror or your API at exactly midnight. The classic cron answer is to hash the hostname into a minute offset in configuration management, which works and which every team reinvents.

systemd has two directives here and they pull in opposite directions, which is the part people get wrong:

• RandomizedDelaySec= spreads timers apart. It adds a delay between zero and the value you set, so identical schedules on many hosts land at different moments.
• AccuracySec= pulls timers together. It lets systemd shift the firing time within a window to coalesce wakeups and save power. It defaults to one minute, which is why a timer set to 02:00:00 often fires a few seconds late.

If you want a timer to fire close to the stated second, lower AccuracySec=. If you want to stagger a fleet, raise RandomizedDelaySec=. Setting only AccuracySec= to a large value and expecting it to spread load is the mistake, because coalescing is the opposite of spreading.

A job that eats the box

Cron has no answer here beyond nice and ionice in front of the command, and a script that hangs forever will hang forever. A systemd service is a cgroup, so the limits are declarative and enforced:

[Service]
Type=oneshot
MemoryMax=512M
CPUQuota=40%
IOWeight=20
TimeoutStartSec=30min
OnFailure=notify-failure@%n.service
ExecStart=/usr/local/bin/reindex.sh

The two directives that earn their keep on a shared box are TimeoutStartSec=, which kills a oneshot job that has hung, and OnFailure=, which starts another unit when this one exits non-zero. That second one is how you get a scheduled job to page you without wiring an alerting call into every script.

The same job written both ways

A nightly database dump that must not overlap, must not run before the database is up, and must catch up if the host was rebooted. In cron:

0 2 * * * /usr/bin/flock -n /var/lock/db-backup.lock /usr/local/bin/db-backup.sh >> /var/log/db-backup.log 2>&1

That handles overlap and captures output. It does not handle “the database wasn’t ready” or “the host was down”, and the log file is now your problem to rotate.

The systemd version is two files. First the service, which is the job:

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly application database dump
After=network-online.target mariadb.service
Wants=network-online.target

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
TimeoutStartSec=45min

Then the timer, which is only the schedule:

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run the nightly database dump

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

Because the two units share a base name, you do not need a Unit= line. systemd pairs db-backup.timer with db-backup.service by convention. Add Unit= only when the names differ.

Then the part people skip, which is verifying it before walking away:

# Check the units parse before you enable anything
systemd-analyze verify /etc/systemd/system/db-backup.*

# Confirm the calendar expression means what you think
systemd-analyze calendar --iterations=5 "*-*-* 02:00:00"

systemctl daemon-reload
systemctl enable --now db-backup.timer

# Next and last elapse for this timer
systemctl list-timers db-backup.timer

# Run it now, out of band, without touching the schedule
systemctl start db-backup.service
journalctl -u db-backup.service -n 50 --no-pager

Note that you enable the timer, not the service. Enabling the service instead would try to run the job at every boot, which is a memorable way to learn the difference.

How I’d decide

Work down this list and stop at the first line that applies. It resolves most cases in under a minute.

1. The host might not be systemd, now or later. Use cron. Portability beats every other consideration on this list.
2. The job is inside a container. Use neither. Use your orchestrator’s scheduler, or a timer on the host that invokes the container.
3. A missed run matters, or the job must not overlap, or it needs to wait for another service. Use a timer. Every one of those is a directive rather than a wrapper script.
4. The job is heavy: a dump, a reindex, an import, anything that can pin a core or fill memory. Use a timer, for the cgroup limits and the runtime cap.
5. You need to see output after the fact without building a logging path first. Use a timer. The journal is already there.
6. You need sub-minute scheduling. Use a timer with a monotonic directive. Cron’s floor is one minute.
7. Otherwise, cron is fine, and one line beats two files.

The migration question answers itself from that. Do not convert forty working crontab lines in a weekend. Write new scheduled work as timers, convert the jobs that have actually burned you, and leave the rest. Mixing both on one host is normal and supported. Just never schedule the same job in both places, which is easier to do than it sounds when a configuration management run and a manual edit disagree.

Diagnosing a job that never fired

Cron

Establish what actually happened before you theorise. The daemon logs every job it starts, so the log tells you which of three situations you are in: cron never tried, cron tried and the command failed, or cron never saw the entry at all.

# Is the daemon running at all?
systemctl status crond      # RHEL, AlmaLinux, Rocky
systemctl status cron       # Debian, Ubuntu

# What did it start, and when?
journalctl -u crond --since "24 hours ago"
grep CRON /var/log/syslog | tail -20     # Debian family
tail -50 /var/log/cron                   # RHEL family

# Whose crontab are you actually looking at?
crontab -l -u deploy
ls -l /etc/cron.d/

Then work through the usual suspects, roughly in order of how often they turn out to be the cause:

• Wrong crontab. The job is in root’s crontab and you are reading the deploy user’s, or it’s in /etc/cron.d where there is an extra user field before the command.
• PATH, or an unescaped percent sign. Dump what the job really sees with a temporary entry like * * * * * env > /tmp/cron-env.txt 2>&1, then compare it against your shell.
• Permissions. The script is not executable, or the crontab owner cannot read it.
• The schedule is not what you think. When both day-of-month and day-of-week are restricted, cron runs the job when either matches, not both. 0 0 15 * 1 fires on the 15th and on every Monday.
• The command captured no output, so a real error was mailed into the void. Append >> /var/log/myjob.log 2>&1 and wait for the next run.

systemd

# Is the timer loaded and scheduled? --all shows inactive ones too
systemctl list-timers --all

# Timer state, last trigger, next elapse
systemctl status db-backup.timer

# What the job printed, and how it exited
journalctl -u db-backup.service --since "7 days ago"

# Prove the schedule independently of the unit
systemd-analyze calendar --iterations=3 "Mon..Fri *-*-* 06:30:00"

• The timer is not enabled. Creating the files does nothing. It needs daemon-reload and enable --now.
• You enabled the service instead of the timer. Check with systemctl is-enabled on both.
• A user timer with no lingering. Under systemctl --user, the manager stops when the session ends unless you run loginctl enable-linger for that user.
• The calendar expression is valid but wrong. Valid syntax that never matches will not error. systemd-analyze calendar is how you find out.
• The service failed before your script ran. A missing User=, an ExecStart path that does not exist, a WorkingDirectory that is not there. The journal names the reason.

Arguments that don’t survive contact

“Cron is deprecated.” It isn’t. It is packaged and supported on every mainstream distribution, and control panels, hosting providers and countless application installers write crontab entries as a matter of course. Nobody is removing it.

“Timers are too complicated for small jobs.” Two files is more typing, not more complexity. The complexity that matters is the wrapper script you write to get locking, logging and a runtime cap out of cron, and how well the next person understands it. Compare like for like.

“The journal means I have logging sorted.” The journal is a ring buffer with a size cap. If nobody set SystemMaxUse= deliberately and nobody ships journal entries anywhere else, your evidence has a retention window you never chose. Check it before you rely on it.

“Switching to timers will stop jobs failing silently.” This is the one I’d push back on hardest. Timers make failures visible. They do not make them noticed. Neither scheduler will ever tell you that a job did not run, because absence is not an event. That needs a dead man’s switch: the job pings a heartbeat endpoint on success, and something external alerts when the ping stops. Healthchecks.io, Cronitor and Better Stack all sell exactly this, and Healthchecks is open source if you would rather self-host it next to your own Grafana. If you already run Prometheus, writing a completion timestamp through the node exporter’s textfile collector and alerting on its age gets you the same property without another vendor.

“Just move everything to timers.” A mass migration of working jobs is a lot of change for no new capability, and every converted job is a chance to fat-finger a path. Convert on cause, not on principle.

Frequently asked questions

Are systemd timers better than cron?

For production jobs on a systemd host, generally yes: logging, dependency ordering, overlap protection, catch-up and resource limits are all built in rather than bolted on. For a one-line job on a box you own, cron’s simplicity is a real advantage and the timer buys you nothing you will use.

Is cron deprecated or being removed?

No. Cron implementations are still packaged and maintained across the major distributions. Some distributions do not install a cron daemon in minimal images by default, which is a packaging decision rather than a deprecation, so check with your package manager before assuming a fresh host has one.

Can cron and systemd timers coexist on the same server?

Yes, and most real servers run both. The cron daemon itself is usually managed as a systemd service. The only rule is that a given job should be scheduled in exactly one place. Double-scheduling is what produces the “why did this run twice” ticket.

How do I see when a systemd timer will next run?

systemctl list-timers shows next and last elapse for all active timers; add --all to include inactive ones. To test an expression without creating a unit, use systemd-analyze calendar "Mon..Fri *-*-* 06:30:00" with --iterations=N for more fire times.

How do I stop a cron job from overlapping itself?

Wrap it in flock -n with a dedicated lock file. The -n flag makes the new run exit straight away instead of queuing. Do not use a PID file you wrote yourself; it will leak a stale lock the first time the job is killed, and then the job stops running forever.

Do systemd timers work inside Docker containers?

Not usefully. A container normally runs a single process without systemd as PID 1, so there is no service manager to own a timer. Schedule from outside the container instead: a host timer that runs the container, or the scheduler your platform provides. Podman is the exception worth knowing, since it can generate systemd units for containers and the host’s systemd drives them.

How do I convert a crontab line to a systemd timer?

Move the command into a Type=oneshot service with an absolute ExecStart, then translate the five cron fields into an OnCalendar= expression of the form DayOfWeek Year-Month-Day Hour:Minute:Second. Verify the expression with systemd-analyze calendar, enable the timer, then remove the crontab line only after you have seen a successful run in the journal.

The one thing worth remembering

Cron vs systemd timers is not a contest between an old tool and a new one. Cron is a scheduler. A systemd timer is a scheduler attached to a supervisor, and the supervisor is what you are actually buying: dependency ordering, overlap protection, catch-up, resource limits, and output that lands somewhere you can read it.

Pick timers when a missed or duplicated run costs you something. Pick cron when it doesn’t. And whichever you pick, add a heartbeat, because the failure that hurts is not the job that crashes. It’s the job that stopped running while everything stayed green.

Need a second pair of eyes on your scheduled jobs?

Scheduling is one of those areas where the problem is rarely the syntax. It’s the job nobody has checked since the person who wrote it left. This is the kind of work I take on:

• Auditing every crontab and timer on a host and telling you which jobs have quietly stopped running
• Converting fragile cron entries to systemd timers with locking, runtime caps and failure handlers, without a big-bang migration
• Fixing the classic “works by hand, fails on schedule” environment and PATH problems, including the percent-sign trap
• Adding heartbeat monitoring and alerting so a missed run pages someone instead of sitting there
• Backup and dump jobs that verify their own output rather than exiting zero on an empty file
• Untangling duplicate schedules left behind by control panels, deploy scripts and configuration management

If something on your server isn’t running when it should, send me the crontab line or the unit file and the journal output around the time it should have fired. That’s usually enough to tell you what’s wrong before we talk about anything else.

The post Cron vs systemd Timers: The Failure Modes That Decide It appeared first on John Nessime.

You go back through the logs and there it is, four hundred lines up, in a step that reported success: a download that returned an HTML error page, got piped into tar, and produced an archive containing nothing useful. The shell reported the exit status of tar, not of curl. The build carried on. The pipeline was never lying on purpose. It just never asked the right question.

Two things go wrong with pipelines, and only one of them gets talked about. The famous one is slow feedback: thirty-four minutes of build and integration tests before a linter tells you a file is missing a trailing newline. The expensive one is a pipeline that passes when it should have stopped. A fail fast pipeline fixes the first problem. Making it explain itself fixes the second, and that is the half most teams skip.

This post covers both: how to order stages so real failures surface in seconds, how to stop the shell from hiding non-zero exits, how to design failure output that hands the next engineer the fix, and the troubleshooting steps for when a pipeline stubbornly refuses to fail at the right moment.

The failure that actually bites: a pipeline that lies

Start here, because everything else is optimisation on top. A slow pipeline wastes your afternoon. A silently passing pipeline ships a broken artifact and burns a day of debugging in production, usually while somebody is on a call with a customer.

The shell is where most of this happens. Bash gives you three defaults that are wrong for CI, and one line fixes all three.

#!/usr/bin/env bash
set -euo pipefail

# Without pipefail this line exits 0 even when curl fails,
# because the shell reports the status of the LAST command
# in the pipe, which is tar.
curl -sSL https://example.com/artifact.tar.gz | tar -xz

What each flag actually protects

• -e stops the script when a command exits non-zero. Without it, a failed database migration is followed by a successful echo, and the script exits 0 because the last command succeeded.
• -u turns an unset variable into an error. This is the one that saves you from rm -rf "$PREFIX/build" expanding to rm -rf "/build" when someone renames an environment variable in the pipeline settings and forgets the script.
• -o pipefail makes a pipeline return the first non-zero status instead of the last one. Every curl | tar, cat | jq, and docker logs | grep in your scripts is currently reporting only the exit code of the thing on the right.

The exit status is only half the story though. Plenty of tools return 0 while doing something useless, and curl is the classic offender.

# curl exits 0 here even when the server returns 500,
# and you end up untarring an HTML error page.
curl -sSL "$ARTIFACT_URL" | tar -xz

# --fail          makes curl exit non-zero on 4xx and 5xx
# --show-error    keeps the reason visible even with --silent
# --location      follows redirects instead of saving a 302 body
curl --fail --silent --show-error --location "$ARTIFACT_URL" | tar -xz

Now the download failing stops the build, and pipefail makes sure that failure propagates past the pipe. The first fixes the check, the second makes sure the check is heard.

Where set -e quietly gives up

This trips people up constantly. set -e is suspended whenever a command’s exit status is being consumed by something else. It is not a safety net, it is a default.

set -e

# -e does NOT stop the script here.
# The if statement consumes the exit status.
if grep -q "TODO" src/main.go; then
  echo "found"
fi

# -e does NOT stop the script here either.
# The left side of || is explicitly allowed to fail.
run_migrations || echo "migrations failed, continuing anyway"

# -e DOES stop the script here.
run_migrations

The same trap applies inside containers. On Debian-based images /bin/sh is dash, which has no pipefail option at all, so putting set -o pipefail inside a RUN line fails outright. Set the shell at the top of the Dockerfile instead. Hadolint flags this as DL4006 and it is worth wiring into your static stage.

# Set this before any RUN instruction that contains a pipe.
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl --fail --silent --show-error --location 
      https://example.com/tool.tar.gz | tar -xz -C /usr/local/bin

Order stages by cost, not by category

Most pipelines are laid out the way the team thinks about work: tests, then build, then security, then deploy. That grouping is comfortable and it is the wrong axis. Order by two things instead, cost and failure probability.

The rule I use: every stage should be cheaper than the one after it and more likely to fail than the one after it. If a check catches ten percent of your broken commits and takes fifteen seconds, it goes in front of the check that catches two percent and takes nine minutes. That is the whole design principle behind a fail fast pipeline.

1. Syntax and static analysis. shellcheck, hadolint, yamllint, terraform validate, tflint, formatter checks, type checks. Seconds, no dependencies, no network. This stage catches an embarrassing share of broken pushes.
2. Unit tests. No database, no containers, no network. If they need any of those, they are integration tests and belong further down.
3. Secrets and dependency scanning. Fast, and a leaked credential is worth stopping before you build an image around it.
4. Build. This is where you start spending real minutes and cache space.
5. Integration and end-to-end. Slow, flaky, expensive to run. Everything above should have already caught the boring failures.
6. Deploy and post-deploy verification.

A note on the trade-off, because this ordering costs something. Splitting the fast checks into their own stage adds a checkout and a container pull before your unit tests even start. On a busy shared runner that overhead is real. If your static stage takes twelve seconds of work and forty seconds of setup, merge it into the test job and run it as the first step instead. The principle is failure order, not job count.

The other thing that keeps the fast lane fast is caching, and it is the piece people get wrong in the opposite direction: they cache so aggressively that a stale dependency tree hides a genuine break. Cache the package manager’s download directory, not the resolved dependency tree, and key the cache on the lockfile hash.

Let the platform kill work you already know is doomed

Ordering stages is only useful if the platform actually stops when something breaks. Three settings do most of the work, and they exist under different names on every runner.

GitHub Actions

concurrency:
  # A new push to the same branch cancels the run that is
  # already in flight. Nobody needs results for a commit
  # that has been superseded.
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 5          # a hung job is worse than a failed one
    steps:
      - uses: actions/checkout@v4   # pin to a full commit SHA in production
      - run: shellcheck scripts/*.sh
      - run: hadolint Dockerfile

  test:
    needs: static               # will not start until static passes
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false          # see note below
      matrix:
        suite: [unit, contract]
    steps:
      - uses: actions/checkout@v4
      - run: make test-${{ matrix.suite }}

That fail-fast: false looks like it contradicts the whole post, so it is worth explaining. Matrix fail-fast defaults to true: one leg fails, the rest get cancelled. That is right when the legs are redundant and you only need to know that something broke. It is wrong when the legs tell you different things. If your matrix is three runtime versions and only one fails, cancelling the others hides whether this is a version-specific problem or a general one, and you will burn a rerun finding out. Fail fast on the pipeline, not on the diagnosis.

timeout-minutes is the setting people skip and regret. A job waiting on an interactive prompt, a dead package mirror, or a container that never becomes healthy will sit there consuming a runner until someone notices. Set a timeout on every job, generous enough that a slow day does not trip it and tight enough that a hang gets caught inside one coffee break.

GitLab CI

stages: [static, test, build, deploy]

default:
  interruptible: true     # allows auto-cancel when a newer pipeline starts
  retry:
    max: 2                # capped at 2 by GitLab, so 3 attempts total
    when:
      # Retry only on infrastructure problems.
      # Never retry script_failure: that is your code failing,
      # and retrying it just hides a flaky test.
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

lint:
  stage: static
  timeout: 5m
  script:
    - shellcheck scripts/*.sh

integration:
  stage: test
  needs: [lint]           # DAG: starts as soon as lint passes
  script:
    - make integration

The retry: when list is the important bit and it is the one people get lazy about. A bare retry: 2 retries on everything including a genuine test failure, which converts an intermittent bug into an invisible one. Scope retries to the failure classes that are genuinely the infrastructure’s fault. Jenkins has failFast true for parallel stages, Buildkite and CircleCI have their own equivalents, and the reasoning transfers cleanly.

A failure should hand you the answer, not the search

Here is the test I apply to a pipeline failure. Can someone who did not write the change, at an inconvenient hour, fix it from the failure output alone without opening the source? If not, the failure message is incomplete.

Four things need to be in there:

• What failed, named precisely. Not “build failed” but the command and its exit code.
• Why, in terms of the input rather than the internals. “lockfile is out of date with package.json”, not “npm ci exited 1”.
• Where, meaning the file, the line, or the config key that caused it.
• What to do next. One concrete action, even if it is just a runbook link.

In shell, a small helper plus an ERR trap gets you most of the way there.

#!/usr/bin/env bash
set -euo pipefail

fail() {
  echo "ERROR: $1" >&2
  echo "  fix: $2" >&2
  exit 1
}

# Catches anything you did not write an explicit check for,
# and tells you the line and the command that blew up.
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

[[ -f config/app.yaml ]] || fail 
  "config/app.yaml is missing" 
  "copy config/app.yaml.example and set DATABASE_URL"

[[ -n "${IMAGE_TAG:-}" ]] || fail 
  "IMAGE_TAG is empty" 
  "check the tag job output; it sets IMAGE_TAG from the git SHA"

Note ${IMAGE_TAG:-} rather than $IMAGE_TAG. Under set -u the bare form aborts with a shell error before your nice message ever prints, which is technically correct and practically useless.

On GitHub Actions you can go one step further and attach the error to the file and line, so it shows up as an annotation on the pull request instead of something to be dug out of a log.

# Renders as a red annotation on that exact line in the diff view.
echo "::error file=deploy/values.yaml,line=42::image tag is empty"

# Visible, but does not fail the job.
echo "::warning file=Dockerfile,line=7::apt cache not cleaned"

Two more habits worth building. Upload logs, test reports, and screenshots as artifacts on failure, conditionally, so the evidence survives the job that produced it. And emit test results as JUnit XML so the platform renders a list of failed test names instead of asking people to scroll. Both are cheap and both cut the time between “it broke” and “I know why” more than any clever retry logic will.

Exit codes are a contract, so use more than one

Most scripts have two states: 0 and “something went wrong”. That collapses distinctions the pipeline needs. Terraform gets this right and it is a good model to copy.

# -detailed-exitcode changes the meaning of the exit code:
#   0 = succeeded, no changes
#   1 = error
#   2 = succeeded, changes present
set +e
terraform plan -detailed-exitcode -out=tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift. Skipping apply." ;;
  2) echo "Changes detected. Holding for approval." ;;
  *) echo "Plan failed. See log above." >&2; exit "$code" ;;
esac

The set +e around it is deliberate: with -e active, exit code 2 would kill the script before you got to interpret it. This is one of the few places where turning the flag off is the correct move, and it is worth a comment in the file explaining why so nobody “fixes” it later.

Do the same in your own scripts. Reserve a code for “the thing you asked for is not possible” versus “I could not reach the API” versus “your input is wrong”. The pipeline can then retry one and refuse to retry the others, which is the difference between a retry that absorbs a runner hiccup and a retry that hides a race condition for six months.

Gate on what you care about, not what is easy to check

This is the deploy-stage version of the same lie. Plenty of commands confirm that a request was accepted, not that the thing you wanted happened.

# Returns 0 the moment the API server accepts the manifest,
# even if every pod goes into CrashLoopBackOff a second later.
kubectl apply -f k8s/

# This is the step that actually waits for the rollout,
# and fails the job if it does not converge.
kubectl rollout status deployment/api --timeout=180s

# And this proves the application answers a request,
# not just that a pod reports Ready.
curl --fail --silent --show-error https://api.example.com/healthz

Run that last check against the public hostname, through your CDN, rather than against the pod directly. If you front the app with Cloudflare or similar, a smoke test that skips the edge will happily pass while real users hit a cached error page or a misrouted origin. Same reasoning applies to the origin host itself: if the app runs on a VPS from a provider like InterServer behind a reverse proxy, verify through the proxy, because that is the path traffic takes.

Then track how long each stage takes over time. Pipeline duration and failure rate per stage are perfectly ordinary metrics to ship into Prometheus and graph in Grafana, or into whatever platform you already use for application monitoring. When the fast lane creeps from forty seconds to four minutes, that is a slow regression nobody will notice from a single run.

Troubleshooting

The job passed but the artifact is wrong

Look for pipes first. Grep the scripts for | and check whether pipefail is set in that shell. Then check for tools that return 0 on failure: curl without --fail, docker run where the entrypoint swallows errors, anything wrapped in || true that somebody added during an incident and never removed.

set -e is set but the script keeps going

The failing command is almost certainly inside a condition, on the left of || or &&, or in a function that is itself being called from an if. Add set -x temporarily and read the trace to see exactly where control flow continued. Then use an explicit check with your fail helper rather than relying on -e at that point.

Works locally, fails in CI (or the reverse)

Usually a shell difference. Your terminal runs bash or zsh, the runner may execute the script with sh. Add an explicit shebang, and on GitHub Actions set the shell on the step or at the job level rather than assuming. Inside containers, remember dash has no pipefail and no [[ ]].

The job hangs until the platform kills it

Something is waiting on stdin. Package managers prompting for confirmation, ssh asking about a host key, a database client waiting for a password. Add the non-interactive flags, set CI=true where tools respect it, and put a timeout on the job so the next occurrence fails in ten minutes instead of sixty.

Fails on the first run, passes on rerun

Resist the retry. Log the failure with enough context to identify it later, quarantine the test into a non-blocking job if it is genuinely blocking the team, and give it an owner and a date. A blanket retry turns a flaky test into a permanent tax on everyone’s confidence in the pipeline.

The failure output is a wall of noise

Suppress progress bars and spinners in CI, they render as thousands of unreadable lines. Then print a short summary block at the very end of the job, after the noise, listing what failed and what to do. People read the bottom of a log first.

Common mistakes

• Writing set -e without -o pipefail and assuming the script is covered.
• Using || true to silence a step that fails intermittently, instead of finding out why.
• Running the slowest, flakiest checks before the ones that take fifteen seconds.
• Retrying on every failure class rather than only infrastructure faults.
• No job timeouts, so a hang consumes a runner until someone notices.
• Treating kubectl apply or a deploy API call as proof the deploy worked.
• Failure messages that name the internal function that threw, and nothing the reader can act on.
• Caching so aggressively that a stale dependency tree hides a real break.
• Discarding logs and test reports when the job fails, which is exactly when you need them.
• Turning off matrix fail-fast everywhere by reflex, so every broken commit costs the full matrix.

Best practices

• Put set -euo pipefail at the top of every shell script the pipeline runs, and set the Dockerfile SHELL before any piped RUN.
• Order stages so each one is cheaper and more likely to fail than the next.
• Give every job a timeout, and cancel superseded runs on the same branch.
• Use explicit checks with actionable messages instead of relying on -e to catch everything.
• Distinguish exit codes so the pipeline can react differently to different failures.
• Verify deploys with a rollout status check plus a real HTTP request through the public path.
• Upload artifacts and structured test reports on failure.
• Pin action and image references to immutable digests so a green pipeline stays reproducible.
• Track stage duration and failure rate over time, the same way you track application metrics.
• Fix the failure message the first time somebody has to ask what it means.

FAQ

What does “fail fast” actually mean in a CI/CD pipeline?

Two things. First, the pipeline stops as soon as it knows the result will be a failure, rather than finishing work whose outcome no longer matters. Second, the cheapest checks that are most likely to catch a problem run first, so the average failure surfaces in seconds. The second half is a design choice about ordering, not a platform setting.

Is set -e enough on its own?

No. It is suspended inside conditions, on the left side of && and ||, and after !. It also does nothing about pipes, which is why -o pipefail matters, or about commands that return 0 while failing, which is why explicit checks matter. Use it as a baseline, not as your only line of defence.

Should I disable matrix fail-fast?

Only when the legs give you different diagnostic information. If the matrix is runtime versions or operating systems and you need to know whether the break is universal or specific, let all legs run. If the legs are interchangeable shards of the same test suite, leave fail-fast on and save the compute.

How do I stop a pipeline from passing when a step really failed?

Work through three things in order. Set pipefail so pipes propagate failure. Add the flags that make tools return non-zero on failure, such as curl --fail. Then replace implicit trust with explicit assertions: after a deploy, check that the rollout converged and that the service answers a request. Most silent passes come from one of those three gaps.

Are automatic retries a good idea?

Only for failures that are the infrastructure’s fault: runner crashes, image pull errors, API timeouts from the CI platform itself. Never retry a script failure by default. And make retries visible, because a job that quietly succeeds on the third attempt is telling you something important that nobody sees.

How long should a fast feedback stage take?

Short enough that an engineer waits for it rather than switching context. Somewhere under a couple of minutes for the combined lint, format, and unit stage is a reasonable target for most codebases. Past that, people push and walk away, and you lose the benefit of fast feedback even though the pipeline is technically quick.

Does this apply to self-hosted runners too?

All of it, and timeouts matter more. On hosted runners a stuck job eventually hits the platform’s own ceiling. On your own machine it holds a slot until you intervene, so a hung job blocks the queue for everyone. Set explicit timeouts and make sure the runner cleans up its workspace between jobs.

The one thing to take away

Speed is the part everyone optimises and honesty is the part that actually costs money. A fail fast pipeline that finishes in ninety seconds is worth very little if one of those seconds contains a swallowed exit code. Fix the lying first: pipefail, tools that return non-zero when they fail, and assertions that check the outcome rather than the request. Then reorder for speed.

And judge every failure by one question. Could someone else fix this from the output alone, without reading the source? If the answer is no, the pipeline has not finished its job, no matter how quickly it went red.

Need help fixing a pipeline that lies to you?

If any of this sounds like your CI, this is the kind of work I take on. Typical engagements look like:

• Auditing an existing GitHub Actions, GitLab CI, or Jenkins pipeline for silently swallowed failures and false-green stages.
• Restructuring stage order and caching so the common failure surfaces in seconds instead of after the build.
• Hardening deploy verification: rollout gates, smoke tests through the real public path, and automatic rollback on a failed health check.
• Rewriting pipeline shell scripts with proper error handling, exit code contracts, and failure messages an on-call engineer can act on.
• Wiring pipeline duration and failure metrics into Prometheus and Grafana so regressions get caught early.

Send me the workflow file and a link to a failing run, and I will tell you what I would change first.

The post Green Check, Broken Build: How to Build a Fail Fast Pipeline That Explains Why appeared first on John Nessime.