That is the moment most people properly meet the command line, usually at the worst possible time. The dashboard is fine for editing a post. It is a bad tool for anything that touches a thousand rows, twelve plugins, or two servers at once.

This is a working list of the WP-CLI commands that save hours, grouped by the job that eats the hours rather than by command name. Each section leads with the failure mode, because with a couple of these the command finishes, prints Success, and leaves you with a subtly broken database that nobody notices for a week.

The migration hour: search-replace, and the one that bites back

Changing a domain by running SQL against wp_posts is how sites break. WordPress stores a great deal of configuration as PHP serialized arrays, and serialized strings carry their own byte length. Change https://old.example.com to https://new.example.org with a plain UPDATE ... REPLACE() and the length prefix no longer matches the string. PHP refuses to unserialize the value, returns false, and the widget or page-builder section silently renders nothing.

The whole point of wp search-replace is that it unserializes, replaces, and reserializes, so the lengths stay correct. Run it in dry-run mode first, every single time:

wp search-replace 'https://old.example.com' 'https://new.example.org' 
  --all-tables-with-prefix 
  --skip-columns=guid 
  --precise 
  --report-changed-only 
  --dry-run

What each flag is actually doing:

• --dry-run runs the whole operation and prints the report without writing. The report has a “Type” column: PHP means the value was serialized and handled properly, SQL means a plain string replacement. If you expected hundreds of changes and see zero, your search string is wrong, not the tool.
• --all-tables-with-prefix widens the scope to every table sharing your prefix, not just the tables registered with $wpdb. Plugins that create their own tables live here. --all-tables goes wider still and will happily rewrite tables belonging to a completely different application sharing the database, so reach for the prefix version first.
• --skip-columns=guid is the one people leave off. The guid column is a permanent identifier for feed readers, not a URL to be followed. Rewriting it makes every historical post look brand new to anything that has subscribed to your feed.
• --precise forces PHP-side serialization handling on every column instead of letting MySQL do the simple cases. It is slower. On a large database it is noticeably slower. It is also the difference between “probably fine” and “definitely correct”.
• --report-changed-only trims the report to tables that actually changed, which makes the output readable instead of a wall of zeros.

Read the dry-run output. When the numbers look right, run the identical command without --dry-run.

The step everyone forgets

You changed the database. Every cache layer still holds the old values. On a site with Redis or Memcached behind a persistent object cache drop-in, the old URLs will keep being served from memory and you will spend twenty minutes convinced the replacement failed.

wp cache flush
wp transient delete --expired
wp rewrite flush

Be aware that on multisite with a shared persistent object cache, wp cache flush can clear entries for every site on the network, not just the one you are working on. Then purge the page cache and the CDN through their own controls. If you are on Cloudflare, the database change means nothing until the edge cache is purged too.

Finally, re-run the dry-run. It should report zero remaining replacements. If it does not, you have a second URL format hiding somewhere: protocol-relative //old.example.com, or an escaped form like https://old.example.com inside JSON stored in an option. Run a second pass for each shape you find.

The plugin conflict hour: bisect without touching the dashboard

Classic scenario: something fatals, the admin is white, and the usual advice is “deactivate all plugins and switch to a default theme.” On a production site that means real downtime while you click through a list.

WP-CLI has two global parameters that skip loading plugins and themes for the duration of a single command. Nothing is deactivated. Nothing is written. The site keeps serving visitors exactly as it was.

# Does WP-CLI work at all with nothing loaded?
wp --skip-plugins --skip-themes option get siteurl

# Is it the theme?
wp --skip-themes option get siteurl

# Skip one specific plugin
wp --skip-plugins=akismet option get siteurl

If the first command works and the unmodified one does not, the fault is in your plugins or theme, not in WP-CLI or WordPress core. From there you can walk the active list one at a time. This one-liner from the WP-CLI handbook does the walk for you:

wp plugin list --field=name --status=active --skip-plugins 
  | xargs -n1 -I % wp --skip-plugins=% plugin get % --field=name

The inner --skip-plugins on the listing command matters. Without it, a plugin broken badly enough to crash the bootstrap will also crash the command that is trying to build the list, and you get nothing.

The honest limitation: skipping a plugin at the CLI does not reproduce browser-side behaviour. If the bug only appears for logged-in users on a specific admin screen, this narrows the field but will not fully confirm the culprit. It still turns a thirty-minute click-through into about two minutes of scrolling output.

The silent-cron hour: scheduled jobs that quietly stopped

This is the invisible failure. Nobody opens a ticket saying “cron is broken.” They open a ticket saying backups have not run since last month, or scheduled posts are stuck in the future, or order emails stopped going out. WP-Cron is not a scheduler. It is a queue that gets drained when someone loads a page, which means a low-traffic site drains it rarely and a fully page-cached site may never drain it at all.

# Can the site reach its own wp-cron.php?
wp cron test

# What is scheduled, and how overdue is it?
wp cron event list

# Drain everything that is due, right now, in this process
wp cron event run --due-now

In wp cron event list, the column to read is next_run_relative. If events are showing as long overdue, the queue is not being drained and you have found your problem.

The fix is to stop relying on page loads. Disable the loopback trigger in wp-config.php, above the “stop editing” line:

define( 'DISABLE_WP_CRON', true );

Then add a real system cron entry for the user that owns the site files:

* * * * * /usr/local/bin/wp --path=/var/www/example.com cron event run --due-now --quiet

Why the CLI form and not a curl of wp-cron.php: wp cron event run executes the events in the current PHP process. No HTTP loopback is involved, so it cannot be defeated by a firewall rule, a reverse proxy, HTTP authentication on staging, or a host that rate-limits self-requests. Errors surface in the cron mail or the log you redirect to, instead of vanishing into a request nobody reads.

One warning. Set DISABLE_WP_CRON and then forget the crontab entry, and you have upgraded an unreliable scheduler into one that never runs at all. Verify with wp cron event list a few minutes later before you close the ticket.

The bulk-edit hour: piping IDs instead of clicking pages

The pattern that does the real work here is --format=ids feeding another command. Once that clicks, most “there is no plugin for this” tasks become one line.

# How many revisions are we carrying?
wp post list --post_type=revision --format=count

# Delete them, skipping the trash
wp post delete $(wp post list --post_type=revision --format=ids) --force

# Every administrator on the site, in one table
wp user list --role=administrator --fields=ID,user_login,user_email

# Regenerate only the thumbnail sizes that are missing
wp media regenerate --only-missing --yes

Two things to know before you run the delete. It is irreversible, so take a database export first. And if the inner command returns nothing because there are no revisions, the outer command gets no arguments and will error rather than doing nothing quietly. That is a harmless failure, but it looks alarming the first time.

Revisions come back unless you cap them. Adding define( 'WP_POST_REVISIONS', 5 ); to wp-config.php keeps enough history to recover from a bad edit without letting wp_posts grow without limit.

For anything you want to script rather than read, add --format=json and pipe it into jq. Every list command in WP-CLI supports it, which makes the whole tool usable as a data source for monitoring and reporting rather than just an admin replacement.

The “is this thing compromised?” hour: checksums

A site is behaving oddly, there is a suspicious redirect, and the first instinct is to install a security scanner. Before that, run the two commands that compare what is on disk against the official hashes on WordPress.org.

wp core verify-checksums
wp plugin verify-checksums --all

These download the published checksums for your exact version and locale and report every core or plugin file that has been modified, plus files that should not exist at all. The core command deliberately avoids loading WordPress, so it still works when the site itself is too broken to boot.

Know the blind spots before you trust a clean result:

• Only plugins hosted on the WordPress.org directory can be verified. Premium plugins, custom code and anything from a private repository have no published source of truth and will be skipped or warned about.
• Nothing in wp-content/uploads is covered. A dropped PHP file in an uploads subdirectory is a common backdoor and checksums will never see it.
• Themes are not covered by an equivalent bundled command.
• Getting a locale or version mismatch produces alarming warnings that are not real findings. Pass --version and --locale explicitly if the output looks wrong.

A clean checksum run is not proof the site is clean. A dirty one is proof it is not, which is worth a great deal in the first five minutes of an incident.

The database hour: find the bloat before you optimise anything

“The site is slow” is not actionable. “One table is carrying most of the database” is. Start by looking, not by running an optimiser.

# Size per table, sorted by whatever is worst
wp db size --tables --human-readable

# Take an export before touching anything
wp db export ~/backups/example-com.sql

# Run a query and strip the ASCII table borders
wp db query 'SELECT option_value FROM wp_options WHERE option_name="home"' --skip-column-names

In practice the offenders are nearly always wp_options stuffed with autoloaded rows and orphaned transients, wp_postmeta from a plugin that never cleans up after itself, or wp_posts full of revisions. Autoloaded options are the sneaky one, because every row marked for autoload is loaded on every request, including requests that will never look at it.

Two clean-up commands worth knowing, and the difference between them:

wp transient delete --expired
wp transient delete --all

The first removes only what has already expired and is safe to run on a schedule. The second removes everything, including live cache entries, which forces WordPress to rebuild them on the next requests. That is safe but produces a short performance dip, so it belongs in a maintenance window rather than in a cron job.

Note that on a site with a persistent object cache drop-in installed, transients bypass the database entirely and wrap the object cache instead. If wp transient delete --all reports nothing, that is usually the reason and not a failure.

The setup that makes everything above faster

If you manage more than one site, the biggest single time saving is not a command at all. It is aliases. Put a wp-cli.yml in a project directory, or ~/.wp-cli/config.yml for a global one:

@production:
  ssh: deploy@example.com/var/www/example.com
@staging:
  ssh: deploy@staging.example.com/var/www/staging

Now you run commands against a remote site from your own machine, without an SSH session:

wp @staging plugin list --status=active
wp @production cron event list

This needs WP-CLI installed on the remote host, and it needs your host to actually give you SSH. That varies more than people expect. A plain VPS from a provider like InterServer or DigitalOcean gives you root and you install what you like. Managed WordPress hosts differ: some ship WP-CLI preinstalled and hand you an SSH key immediately, others restrict which commands you can run, and a few give you no shell at all on entry-level plans. Worth checking before you build a workflow that depends on it.

Two optional packages are worth the install if you do maintenance work regularly:

wp package install wp-cli/doctor-command
wp package install wp-cli/profile-command

wp doctor check --all
wp profile stage

wp doctor runs a set of health checks, including one for autoloaded options size, and gives you a pass or warn per check. wp profile breaks a request into stages so you can see where the time is going before you start guessing.

The honest trade-off: these are separate packages, not bundled commands, and they get less attention than the core set. If you are working across client servers you do not control, installing packages everywhere is friction you may not want. On your own infrastructure they earn their place. For continuous visibility rather than point-in-time checks you still want proper monitoring, whether that is a self-hosted Prometheus and Grafana stack or a hosted uptime service.

Troubleshooting: when WP-CLI itself will not run

Before blaming the tool, run wp cli info. It prints the PHP binary in use, the PHP version, the loaded php.ini, and the WP-CLI version. Most problems are visible in that output.

• “This does not seem to be a WordPress installation.” You are in the wrong directory, or the document root is elsewhere. Pass --path=/full/path/to/wordpress rather than guessing.
• Wrong PHP version. On a control panel server the shell PHP is often not the PHP the site runs on. Check the binary in wp cli info, and invoke the right one explicitly if they differ, for example /usr/local/php83/bin/php $(which wp).
• Memory exhaustion on large operations. The CLI uses its own memory limit, not the web one. Raise it for a single command with php -d memory_limit=512M $(which wp) ... instead of editing configuration globally.
• A fatal error the moment you type anything. Confirm with wp --skip-plugins --skip-themes cli info. If that works, the fault is in site code.
• A warning about running as root. WP-CLI is telling you that files created by this command will be owned by root and the web server may not be able to write to them later. --allow-root silences the warning without fixing the ownership problem. Run as the site user instead, unless you are inside a container where root is the normal case.

Common mistakes

• Running search-replace without --dry-run because you have done it before and it worked.
• Reaching for --all-tables by reflex when --all-tables-with-prefix is what you meant, on a server where several sites share one database.
• Forgetting to flush the object cache and CDN after a database write, then re-running the replacement several times chasing a ghost.
• Setting DISABLE_WP_CRON without adding the system cron entry.
• Treating a clean verify-checksums result as proof that a site is not compromised.
• Running destructive commands over SSH without screen or tmux, then losing the connection mid-write on a large table.
• Using --allow-root as a habit and leaving root-owned files scattered through wp-content.

Best practices

1. Export the database before any command that writes. wp db export takes seconds and has saved more afternoons than any other command in this post.
2. Dry-run first, read the report, then repeat the command verbatim without the flag. Do not retype it.
3. Run as the site user, not root, so file ownership stays correct.
4. Use aliases in wp-cli.yml so you cannot run a staging command against production by muscle memory.
5. Verify after the write. Re-run the read-only version of whatever you just did and confirm it reports zero remaining work.
6. Test the restore path, not just the backup. An export you have never imported is a hope, not a backup.
7. Keep WP-CLI current with wp cli update, particularly before working on a site running a recent PHP release.

Frequently asked questions

Is wp search-replace safe to run on a live site?

Safer than raw SQL, because it handles serialized data correctly. Still a write against production. Export the database first, dry-run first, and use --skip-columns=guid. On a large database the operation can hold the site in a slightly inconsistent state for the duration, so a quiet window helps.

Why do my changes not show up after search-replace?

Almost always caching. Run wp cache flush, purge the page cache and the CDN, then reload. If the old value still appears in a dry-run report, you have a second URL format stored somewhere, such as a protocol-relative or escaped variant.

Does WP-CLI need SSH access?

It needs shell access on a machine that can reach the WordPress files and database. That usually means SSH. Some managed hosts provide a browser terminal instead, and some restrict the available commands. On a self-managed VPS you install it yourself and nothing is restricted.

What is the difference between –skip-plugins and deactivating a plugin?

--skip-plugins prevents the plugin from loading for that one command only. Nothing is written to the database and visitors are unaffected. Deactivating changes site state and takes the plugin offline for everyone, which is why the skip flags are the right diagnostic tool on production.

Can I schedule WP-CLI commands with cron?

Yes, and that is the recommended way to run WordPress scheduled events reliably. Use the full path to the wp binary, pass --path, run as the site user, and redirect output so failures are visible rather than silent.

Will verify-checksums detect all malware?

No. It compares core files and WordPress.org-hosted plugins against published hashes. It does not cover themes, premium or custom plugins, or anything in the uploads directory. Treat a failure as a definite finding and a pass as one data point among several.

Which WP-CLI commands should I learn first?

wp db export, wp search-replace --dry-run, wp --skip-plugins, and wp cron event list. Those four cover backup, migration, conflict isolation and the most common silent failure on a WordPress site.

The one thing worth remembering

Most of the WP-CLI commands that save hours do it by turning a job you cannot see into a job you can read. --dry-run shows you the write before it happens. wp cron event list shows you a queue that was failing silently. verify-checksums shows you a file that changed without anyone deciding it should.

The speed is real, but the speed is not the point. The point is that you stop guessing. If you take one habit from this post, take the dry-run: run the read-only version, read the output, then run the identical command for real. That single pattern prevents the majority of the incidents I see caused by command-line work on WordPress.

Need a hand with this?

I work on WordPress infrastructure, and command-line automation is most of what makes that maintainable. Things I can help with directly:

• Domain and host migrations run properly, with staged dry-runs, correct table scoping and a verified rollback before anything is written.
• Replacing WP-Cron with real system cron across a fleet, including alerting when a job stops firing instead of finding out weeks later.
• Writing repeatable WP-CLI maintenance scripts, with proper exit codes and logging, so they can be scheduled rather than remembered.
• Diagnosing slow WordPress sites at the database layer, from autoloaded options and table bloat down to the query causing it.
• Incident triage on a site suspected of being compromised, starting with integrity checks and file-level evidence rather than a scanner plugin.
• Setting up multi-site management workflows with WP-CLI aliases, SSH keys and deployment pipelines that do not depend on anyone opening the dashboard.

If something specific is going wrong, send the actual output. A dry-run report, a wp cli info dump, a wp cron event list table, and I can usually tell you where the problem is before we talk about scope.

The post WP-CLI Commands That Save Hours (And the One That Bites Back) appeared first on John Nessime.

Then it clicks. Somebody logged into wp-admin last month, saw an update notice, clicked it. The plugin got updated on the server and never in git. Last night’s deploy synced the repo over the top and put the old version back, along with the vulnerability the update was patching. No error, no warning. The deploy did exactly what it was told.

That is the failure mode that kills most attempts at Git-based WordPress deployment, and it has nothing to do with your pipeline. WordPress is designed to modify itself. It installs plugins, updates core, writes files, all from the browser. The moment you put it under version control you have two systems that both believe they own the file tree, and they will quietly overwrite each other until somebody notices.

This post covers the whole chain: what belongs in the repo, how to shut the second write path, how configuration differs per environment, which way the database is allowed to travel, and how to make staging safe enough that you can restore production data into it without emailing your customers by accident.

The two things that make WordPress awkward under Git

Everything in this workflow follows from two facts.

State is split across files and a database. Your theme is code. Your posts are data. But so are your plugin settings, your menus, your widget layouts, and your permalink structure, and those live in the database next to the content. Git only manages half your site, and it is not obvious which half a given change landed in.

The two halves travel in opposite directions. Code moves forward, local to staging to production. Content moves backward, production to staging to local. Once a site is live, production is the only authoritative source of orders, comments and posts, and pushing a database upward destroys everything written since your last pull.

Write that rule somewhere the whole team can see it. Almost every catastrophic WordPress deployment story is somebody pushing a database in the wrong direction.

Decide what actually goes in the repo

Three approaches, in increasing order of discipline and payoff. Pick one deliberately rather than drifting into the first.

1. Theme only. The repo contains one custom theme, sometimes a custom plugin. Everything else is installed through wp-admin. Easy to start, and it means you have no reproducible build: rebuilding the site from the repo is impossible.
2. All of wp-content, minus uploads. Themes, plugins and mu-plugins in git; core and media excluded. This is the pragmatic middle and where most teams land. You get reproducible plugin versions and a real diff when a plugin changes.
3. Composer-managed, Bedrock-style. Core, plugins and themes are all declared as dependencies. The repo holds a composer.json and your own code, nothing else. The lockfile is the source of truth and composer install rebuilds the site anywhere.

Option three is the one I reach for on anything with more than one developer, because “which plugin version is production running” becomes a question with an answer. The honest cost: premium plugins that are not on a Composer repository need either a private repository or a manual step, and that friction is real. If most of your plugin stack is commercial, option two is a reasonable place to stop.

One thing to check if you are following an older tutorial. The Composer repository landscape for WordPress plugins shifted recently: WPackagist was acquired by WP Engine, and the Roots team launched WP Packages as an independent alternative with different package naming. Both work. Any guide written before that change will use the older wpackagist-plugin/ prefix, so make sure the repository URL and the prefixes in your composer.json match each other.

The gitignore that matters

# Secrets and machine-specific config. Never committed.
wp-config.php
.env

# Content, not code. This gets pulled down from production,
# it never gets deployed up.
wp-content/uploads/

# Generated at runtime or by the build.
wp-content/cache/
wp-content/upgrade/
wp-content/debug.log
node_modules/
vendor/

Ignoring vendor/ assumes your deploy runs composer install. If your pipeline just rsyncs the repo with no build step, commit it instead, or you will ship a site with no dependencies. Decide which, then be consistent, because half-and-half is how you end up with a missing autoloader at two in the morning.

Close the second write path

This is the step people skip, and it is the one that makes everything else work. If production can still install and update plugins from the browser, your repo is not the source of truth. It is a suggestion.

// wp-config.php on production and staging.

// Removes the ability to install, update or delete plugins and
// themes from the admin. Also disables the built-in file editor,
// so DISALLOW_FILE_EDIT is implied by this one.
define( 'DISALLOW_FILE_MODS', true );

// Stops core from updating itself behind your back, including
// the automatic minor updates that would otherwise change files
// your deploy is about to overwrite.
define( 'AUTOMATIC_UPDATER_DISABLED', true );

Be clear-eyed about the trade-off, because it is a real one. You have just taken away automatic security updates. That is only an improvement if you replace them with something: a scheduled dependency update job, a weekly review of the update list, a bot that opens a pull request when a plugin version changes. If you disable updates and then do nothing, you have made the site less safe, not more controlled.

The update flow now runs through the repo. On your local machine:

# Composer-managed: bump the lockfile, commit it, deploy.
composer update wp-plugin/some-plugin

# wp-content-in-git: let WP-CLI do the download locally,
# then commit the resulting file changes as a normal diff.
wp plugin update some-plugin

The second form is worth appreciating. A plugin update becomes a reviewable diff. When something breaks two days later, git log tells you exactly which files changed and git revert puts them back.

Configuration per environment

Database credentials, salts and API keys differ per environment and none of them belong in git. Keep wp-config.php out of the repo and place it on each server, or use an environment file with a committed .env.example showing the required keys with no values.

Core has a first-class way to tell environments apart, and it is underused:

// Per environment, in wp-config.php.
// Recognised values: 'local', 'development', 'staging', 'production'.
define( 'WP_ENVIRONMENT_TYPE', 'staging' );

<?php
// mu-plugin, committed to the repo, safe on every environment
// because it checks where it is running.

if ( wp_get_environment_type() !== 'production' ) {

    // Keep non-production sites out of search results.
    add_filter( 'pre_option_blog_public', '__return_zero' );

    // Short-circuit wp_mail() entirely. Returning a non-null
    // value stops WordPress before it hands anything to PHPMailer.
    add_filter( 'pre_wp_mail', '__return_false' );
}

That mail filter is the single most valuable line in this post. Restore a production database onto staging, forget this, and the first cron run mails real order confirmations, password resets and abandoned-cart nudges to real customers from a URL that does not work. It is silent until it is very loud.

Moving the database in the allowed direction

Set up WP-CLI aliases once and the rest becomes short commands instead of a runbook nobody follows.

# wp-cli.yml in the project root, committed to the repo.
@staging:
  ssh: deploy@staging.example.com/var/www/staging.example.com
@production:
  ssh: deploy@example.com/var/www/example.com

Now pulling production down to your machine is one pipe. The trailing hyphen means “write the dump to standard output”, and the leading hyphen on the import means “read it from standard input”, so nothing touches disk:

# Back up your local database first, then overwrite it.
wp db export local-backup.sql
wp @production db export - | wp db import -

The URLs inside that dump still point at production, and they are buried in serialized PHP arrays where a plain SQL find-and-replace corrupts the string length prefixes. This is exactly what wp search-replace exists for: it unserializes, replaces, and reserializes properly.

# Always dry-run first and actually read the table list it prints.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid --dry-run

# Then for real.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid

wp cache flush

Two flags worth understanding rather than copying:

• --skip-columns=guid because the guid column is a permanent identifier that feed readers and external systems use to decide whether they have already seen a post. Rewriting it makes subscribers re-receive your entire archive, and you find out from an angry email weeks later.
• --all-tables-with-prefix because plugins create their own tables using your prefix, and those tables are not registered with WordPress. Without this, the replacement misses them and you get half-migrated URLs in places nobody checks.

Uploads are the other half of the content problem. Do not put them in git; a media library will outgrow a repository quickly and every clone pays for it. Sync them separately with rsync when you need them, or skip them entirely locally and let a plugin proxy missing images from production. Offloading media to object storage sidesteps the question completely and is worth considering on any site with a large library.

The deploy itself

Three broad options. git pull on the server is the simplest and the one I would move away from first: it puts a .git directory next to your document root, it has no build step, and a failed pull leaves the site half-updated. Rsync from CI is the pragmatic default. Atomic releases with a symlink swap, the way Deployer or Capistrano work, are the most correct because the site switches versions instantly and rollback is a symlink change.

A workable rsync deploy from GitHub Actions, with staging on merge and production on tag:

name: Deploy
on:
  push:
    branches: [main]     # goes to staging
    tags: ['v*']         # goes to production

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4    # pin to a full SHA in production

      - name: Build
        run: |
          composer install --no-dev --optimize-autoloader
          npm ci
          npm run build

      - name: Sync files
        run: |
          echo "$DEPLOY_KEY" > deploy_key
          chmod 600 deploy_key
          rsync -az --delete 
            -e "ssh -i deploy_key -o StrictHostKeyChecking=accept-new" 
            --exclude-from=.deployignore 
            ./ "$TARGET"
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
          TARGET: ${{ secrets.DEPLOY_TARGET }}

--delete is what makes the target match the source exactly, which is the entire point of a deploy. It is also the flag that will erase your uploads directory if .deployignore is wrong. Test the exclude list against staging before you ever point this at production, and use --dry-run the first time.

Files landing on the server is not the same as a working deploy. Run a post-deploy step over SSH:

#!/usr/bin/env bash
set -euo pipefail

cd /var/www/example.com

# Applies schema changes that a core or plugin version bump needs.
# Safe to run when there is nothing to do.
wp core update-db

# Rewrite rules live in the database, so a plugin that registers
# new routes needs this or you get 404s on the new URLs.
wp rewrite flush

# Options and transients are cached; stale ones survive a file deploy.
wp cache flush

# Prove the site answers before you call the deploy finished.
# --fail makes curl exit non-zero on a 5xx instead of shrugging.
curl --fail --silent --show-error https://example.com/ > /dev/null

If you run PHP-FPM with OPcache, add a reset to that script. Otherwise PHP happily serves the previous version of your code from its compiled cache and you spend twenty minutes convinced the deploy did not run.

Making staging safe to hold real data

Staging is only useful if it mirrors production, and the moment it does it becomes dangerous. Every one of these has bitten somebody:

• Outbound email. Covered above. Kill it at the wp_mail() level rather than trusting a plugin setting somebody can toggle.
• Cron. A restored production database brings production’s scheduled jobs with it. Set DISABLE_WP_CRON on staging and do not add a system cron job unless you specifically need to test scheduling.
• Payment gateways. The database carries live API keys. Overwrite them with test-mode credentials as part of your restore script, not by hand afterwards.
• Search indexing. Filter blog_public as shown, and put HTTP basic auth in front of the whole environment. Duplicate content ranking above your real site is an avoidable embarrassment.
• Personal data. A production restore is real customer data on a server that probably has weaker access controls. Either anonymise user records during the restore or treat staging with the same seriousness as production.

Write the restore as a single script that does all of this in order, so nobody has to remember step four. If it is a checklist in a wiki, it will be skipped.

Troubleshooting

The deploy ran but the site is unchanged

In order: OPcache serving compiled bytecode, a page cache or CDN holding the old response, or rsync excluding the directory you edited. Check them in that order. Confirm the file on disk actually changed before blaming anything upstream of it.

White screen immediately after deploying

Usually a missing autoloader, meaning vendor/ was neither committed nor built. Check the PHP error log rather than guessing. A memory limit lower on the server than locally is the other common cause, and it produces the same blank page.

Site works, admin redirects in a loop

siteurl and home in the options table disagree with the URL you are actually using. Almost always a search-replace that missed. Check with wp option get siteurl and wp option get home, and set them explicitly with wp option update rather than running the replacement again.

Serialized data broken after a migration

Somebody ran sed or a SQL REPLACE on the dump. The length prefix inside each serialized string no longer matches the string, so PHP silently fails to unserialize and widgets, theme options and ACF fields come back empty. Restore from the dump and redo it with wp search-replace. There is no partial recovery worth attempting.

Permalinks return 404 after deploy

Rewrite rules are stored in the database and did not get regenerated. Run wp rewrite flush. If it recurs on every deploy, add it to the post-deploy script permanently.

Plugin reverts to an older version after every deploy

Somebody is still updating in wp-admin. That is not a pipeline bug, it is the second write path being open. Set DISALLOW_FILE_MODS and give the team a documented way to request an update.

Common mistakes

• Committing wp-config.php, and with it your database password and salts.
• Putting wp-content/uploads in the repo, then wondering why clones take ten minutes.
• Pushing a staging database over production because “the content is the same”. It never is.
• Using sed or SQL to swap URLs instead of wp search-replace.
• Replacing the guid column, and re-sending your archive to every RSS subscriber.
• Running rsync --delete against production without testing the exclude list.
• Leaving updates enabled in wp-admin, so the repo and the server drift apart silently.
• Disabling automatic updates without putting a review process in its place.
• Restoring production data onto staging with mail still enabled.
• Forgetting DISABLE_WP_CRON on staging, so scheduled jobs fire twice across two environments.
• Treating “the files copied” as proof the deploy worked, with no smoke test.

Best practices

• Make the repo the only way code reaches a server, and enforce it with DISALLOW_FILE_MODS.
• Keep code moving forward and content moving backward, and state that rule where the team can see it.
• Deploy staging from a branch and production from a tag, so production releases are deliberate.
• Dry-run every search-replace and every rsync the first time.
• Back up the target database immediately before any deploy that touches it.
• Script the staging restore end to end, including mail, cron, keys and anonymisation.
• Finish every deploy with a real HTTP request through the public URL, not a file listing.
• Use environment-aware code with wp_get_environment_type() instead of commented-out blocks.
• Keep uploads out of git and sync or offload them separately.
• Replace automatic updates with a scheduled dependency review, and put it in a calendar.

FAQ

Should WordPress core be in the repository?

Only if you are not managing it another way. Committing core works and makes the repo self-contained, at the cost of noisy diffs on every update. Declaring core as a Composer dependency is cleaner and gives you a pinned version without the noise. What you should not do is leave core unmanaged, so nobody can say which version production runs.

How do I handle the database in a Git workflow?

You do not put it in Git. Git handles code; the database is handled by scripted pulls in one direction. Production to staging to local, with a search-replace on import. The only time a database legitimately goes upward is the initial launch, before there is real content to lose.

What about plugin settings changed in staging?

This is the genuinely hard part and there is no clean answer. Settings live in the database, so they do not deploy. Options are: reapply them by hand in production with a documented checklist, script them as WP-CLI commands in a deploy step, or use a plugin that exports configuration to files. Whichever you pick, write it down, because “I’ll remember” is how a setting gets applied to staging and never production.

Is Bedrock necessary?

No. It is a well-made set of defaults for Composer-managed WordPress with environment-based configuration, and it saves assembling those pieces yourself. Plenty of teams run a perfectly disciplined Git deployment with a standard directory layout. Adopt it if you want the conventions; do not adopt it as a prerequisite.

Do I need three environments?

You need local and production. Staging earns its cost when you have multiple people, a client who signs off on changes, or a site where breakage costs money. On a personal blog it is overhead. On a store, skipping it means production is your test environment.

Can I do this on shared hosting?

Partly. You need SSH and ideally WP-CLI, and plenty of shared plans have neither. Some managed WordPress hosts offer a git push deployment target, which handles the file half for you. If you want the full workflow with your own build steps and post-deploy hooks, a small VPS from a provider like InterServer gives you the SSH access and cron control the workflow assumes.

How do I roll back a bad deploy?

Files are easy: redeploy the previous tag, or swap the symlink back if you use atomic releases. The database is the hard part, which is why you take a dump immediately before any deploy that runs a migration. Rolling files back without rolling the database back can leave you worse off than the broken version, so know which of the two actually changed before you act.

The one thing to remember

Git-based WordPress deployment is not really a pipeline problem. The pipeline is the easy part, and you can build a working one in an afternoon. The hard part is making the repo the only way code reaches a server, and then keeping the database flowing in exactly one direction.

Get those two right and everything else is detail. Get them wrong and you will keep having the same confusing morning, staring at a plugin version that nobody remembers changing.

Want this set up properly on your site?

Retrofitting version control onto a live WordPress site is fiddly work, mostly because you have to do it without downtime and without losing whatever has been changed directly on the server. Things I take on:

• Moving an existing production site into Git without downtime, including reconciling whatever has drifted on the server.
• Converting a site to Composer-managed plugins and core, with a working lockfile and a sane update process.
• Building local, staging and production environments that actually match, with per-environment configuration.
• Deploy pipelines in GitHub Actions or GitLab CI: build, rsync or atomic releases, post-deploy WP-CLI steps, smoke test, rollback path.
• Scripted database refreshes from production to staging with mail, cron, API keys and personal data handled automatically.
• Server setup for the workflow: SSH deploy users, correct file ownership, OPcache resets, cron.

Tell me how your site is hosted and how plugins currently get updated, and I will tell you what the first step should be.

The post Two Sources of Truth: Git-Based WordPress Deployment From Local to Production appeared first on John Nessime.

The line you wrote was real. The file was saved. What you did not know is that your distro puts an Include /etc/ssh/sshd_config.d/*.conf near the top of sshd_config, that sshd uses the first value it finds for any given keyword, and that your cloud provider’s image dropped a 50-cloud-init.conf in that directory setting PasswordAuthentication yes. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.

You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.

That is the thing most guides get wrong, and it is the idea this VPS hardening checklist is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.

This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.

Before you touch anything

Two minutes of preparation that saves an evening.

Find your provider’s console before you need it. Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.

Never close the session you are working in. Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.

Then update the system, because everything else assumes current packages:

sudo apt update && sudo apt upgrade -y
sudo reboot   # if a kernel update landed

1. A real user, and keys instead of passwords

Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.

# On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"

# On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10

Now open a second terminal and confirm ssh deploy@203.0.113.10 works, and that sudo -v works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.

2. SSH config, and why your edit might do nothing

This is the section that actually matters. Two mechanisms silently override what you write.

Drop-in files and first-match-wins

Modern OpenSSH packages ship an Include line near the top of /etc/ssh/sshd_config, which pulls in every .conf file from /etc/ssh/sshd_config.d/ in alphabetical order, before the rest of the main file is read. Combine that with sshd’s rule that the first value found for a keyword wins, and the consequence is specific: your drop-in has to sort earlier than the vendor’s, not later.

Most guides tell you to name it 99-something.conf, which is the convention almost everywhere else in Linux and exactly backwards here.

# /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy

A note on KbdInteractiveAuthentication: older guides and older OpenSSH call this ChallengeResponseAuthentication. Setting only PasswordAuthentication no can still leave a keyboard-interactive path open on some configurations, which is why both are here.

Now the part nobody does. Validate the syntax, then read the effective config:

# Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'

If passwordauthentication yes comes back, your file lost to something else in that directory. Look at what is in there with ls /etc/ssh/sshd_config.d/ and either rename yours to sort earlier or edit the offending file directly.

Socket activation, and the port that will not change

The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: ssh.socket holds the listening port and starts sshd only when a connection arrives. On those systems the Port and ListenAddress directives in sshd_config may be ignored entirely, because systemd owns the socket.

People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:

# -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp

If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under /lib/systemd/system/, which a package update will overwrite:

sudo systemctl edit ssh.socket

# In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222

sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it

Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.

3. Firewall, in the right order

The ordering here is the whole trick. Set the default policy and add your rules before you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.

sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose

Read that status verbose output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.

One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with -p 5432:5432 is reachable from the internet even though ufw says that port is denied. Bind container ports to 127.0.0.1 explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.

A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.

4. Automatic security updates

Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.

sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug

By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.

5. fail2ban, and what it is actually for

Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.

What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials do exist: a mail server, a web application login form, an FTP daemon.

sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd

Check that status output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, journalctl -u fail2ban will tell you why.

Set ignoreip in jail.local to include your office or home address. Banning yourself is a rite of passage but it is avoidable.

6. The boring items that matter more than the exciting ones

Nobody writes blog posts about these, and they cause more real outages than any of the above.

Provider account security

Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.

Time and timezone

sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"

UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.

Swap on small instances

# A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

Backups that leave the machine

Your provider’s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.

Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man’s switch service at the backup job so you find out when it stops running, rather than the day you need it.

Knowing the box is alive

At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.

What I would skip

Not everything in the long hardening guides earns its cost on a single web server.

• Large sysctl hardening blocks pasted from a gist. Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.
• AIDE, auditd and file-integrity monitoring on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.
• Rebuilding OpenSSH’s cipher list. Current defaults are good. Guides that hand you a Ciphers line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.
• Disabling ping. It breaks your own monitoring and hides nothing.

Troubleshooting

Locked out after an SSH change

Provider console, log in as root, and revert. If you cannot remember what you changed, ls -lt /etc/ssh/sshd_config.d/ shows the most recently modified file. Run sshd -t to see whether the config is even valid, since a syntax error stops sshd starting at all.

Key auth rejected, password prompt appears instead

Almost always permissions. The home directory must not be group-writable, ~/.ssh needs 700 and authorized_keys 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run ssh -v to see how far it gets, and check journalctl -u ssh on the server for the actual reason.

The setting I applied is not in effect

sudo sshd -T and search for the keyword. If the effective value differs from your file, something in sshd_config.d/ sorts earlier, or a Match block later in the config is overriding it for your connection.

A service is reachable that the firewall says is blocked

Docker, or another daemon writing netfilter rules directly. Check with sudo iptables -S or sudo nft list ruleset, and always verify from a machine outside the server rather than trusting ufw status.

Nothing obviously wrong, but who has been logging in?

last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"

Common mistakes

• Closing your only SSH session before testing the new configuration from a second one.
• Enabling the firewall before adding the SSH rule.
• Naming your sshd drop-in 99-, so a vendor file that sorts earlier wins.
• Trusting the file you edited instead of running sshd -T.
• Generating the SSH key on the server.
• Changing the SSH port and counting it as a security control.
• Disabling password auth for SSH while leaving a web control panel with a weak password exposed.
• No two-factor authentication on the hosting account that can reset root.
• Treating provider snapshots as backups.
• Installing fail2ban and never checking that it started.
• Publishing container ports without binding them to localhost, then assuming ufw covers it.
• Pasting a sysctl block you cannot explain line by line.

Best practices

• Verify every control after applying it, with a command that reads the running state.
• Keep a second session open and the provider console tested before making SSH changes.
• Key-only authentication, root login disabled, and an explicit AllowUsers list.
• Default-deny inbound firewall, with rules added before the firewall is enabled.
• Automatic security updates on, with a deliberate decision about automatic reboots.
• Two-factor authentication on the hosting account and the registrar.
• Backups off the machine, on separate credentials, and restore-tested at least once.
• UTC everywhere, with clock sync confirmed.
• Monitoring that pages you on disk usage before it pages you on downtime.
• Script the whole build, so the next server is identical and you are not repeating this from memory.

FAQ

What is the single most important step?

Key-only SSH with root login disabled, verified with sshd -T. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.

Should I change the SSH port?

For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.

Do I need fail2ban if password authentication is off?

Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.

Why did my sshd_config change get ignored?

Two likely causes. A file in /etc/ssh/sshd_config.d/ that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is Port or ListenAddress on a socket-activated system, where systemd owns the listener. sshd -T answers the first case and ss -tlnp answers the second.

Is ufw enough, or should I use nftables directly?

ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.

How do I harden a server I did not build?

Audit before you change anything. sshd -T, ufw status verbose, ss -tlnp for unexpected listeners, systemctl list-units --type=service --state=running, and the contents of every authorized_keys on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.

Should this be automated?

Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.

The one thing to remember

Editing a config file is not the same as changing a system’s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.

So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. sshd -T for SSH, ufw status verbose for the firewall, ss -tlnp for what is listening, systemctl status for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.

Want a second pair of eyes on your server?

Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:

• Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.
• Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.
• Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.
• Moving SSH off the public internet onto a private overlay network or bastion.
• Backup strategy that survives the hosting account being compromised, including a tested restore.
• Firewall and container networking, especially where published Docker ports are bypassing the firewall.

Send me the output of sshd -T, ufw status verbose and ss -tlnp, and I will tell you what stands out.

The post You Probably Didn’t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself appeared first on John Nessime.

That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer’s token, pushed a malicious commit, and then retroactively repointed every version tag at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner’s secrets into the build log. On public repositories, those logs are readable by anyone.

Nobody’s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.

A GitHub Actions VPS deployment is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.

Who can actually reach that key

Worth being explicit, because it is a longer list than people expect:

• Anyone with write access to the repository. A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.
• Every action in the job. Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.
• Anyone who can influence what runs. This is why pull_request_target combined with checking out the pull request’s own code is such a well-known foot-gun: it runs the fork’s code with the base repository’s secrets in scope.

The goal is not to make the key unreachable. It is to make a stolen key not worth much.

Decide how the runner authenticates

Three options, and the right one depends on what you already run.

1. A dedicated SSH key in repository secrets. The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.
2. An overlay network. Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.
3. A self-hosted runner on the server. No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.

One thing that sounds appealing and is not practical: allowlisting GitHub’s runner IP ranges at your firewall. The ranges are published through GitHub’s meta API, but they are enormous and they change, and every other GitHub customer’s job runs from inside them. It buys you very little for a lot of maintenance.

Narrow the key until it is boring

Assume the key leaks. What can the holder do? Work backwards from there.

Give it its own user. A deploy account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.

Restrict the key in authorized_keys. SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:

# ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions

The guard script decides what the key is allowed to do. Keep it short enough to read in one go:

#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" >&2
    exit 1
    ;;
esac

Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.

Scope the sudo entry too. The deploy user needs to reload a service, not run arbitrary commands as root:

# visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted deploy user with no sudo is a reasonable stopping point.

Verify the server, not just the key

Almost every tutorial on this topic contains StrictHostKeyChecking=no, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.

# Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com

Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.

The workflow

name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" > ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz > /dev/null

Three details in there worth calling out, because they are the ones people leave out.

-o BatchMode=yes stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.

--delete is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if .deployignore is wrong. Run it once with --dry-run against staging and read the list before you point it at production.

The final curl --fail is the difference between “the files copied” and “the site works”. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.

Third-party actions

There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain ssh and rsync for that reason.

Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.

Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.

Gate production properly

The environment: key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:

• Scoped secrets. The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.
• Required reviewers. The job pauses and waits for a named human to approve it. This is the single control that turns “anyone with write access can deploy” into “anyone with write access can request a deploy”.
• Deployment branch and tag rules. Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.

Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.

For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.

Troubleshooting

Permission denied (publickey)

The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses printf '%sn' rather than echo: it guarantees the newline regardless of what got stored.

After that, check permissions on the server: ~/.ssh must be 700 and authorized_keys 600, owned by the deploy user, and the home directory must not be group-writable. Add -vvv to the ssh command to see how far the handshake gets, and read journalctl -u ssh on the server for the reason, since the client is deliberately vague.

Host key verification failed

Either the known_hosts variable is empty, or the server’s host key genuinely changed. If you rebuilt the server, regenerate it with ssh-keyscan and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.

The job hangs until the timeout

SSH is waiting on a prompt nobody will answer. Add -o BatchMode=yes and it fails immediately with a readable reason instead.

rsync works from my laptop, fails from the runner

If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records $SSH_ORIGINAL_COMMAND, run the job once, and read what rsync actually sent.

Works on push, fails on pull requests

Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for pull_request_target to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.

A secret appeared in the logs

Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.

Common mistakes

• Using an existing personal SSH key instead of generating a dedicated one for the runner.
• Giving the deploy user full sudo, or a sudo rule with a wildcard in it.
• Disabling host key checking with StrictHostKeyChecking=no.
• Referencing third-party actions by tag in the job that holds your production key.
• Interpolating ${{ secrets.X }} directly into a run: script instead of passing it through env:.
• Leaving default write permissions on a job that only needs to read.
• Using cancel-in-progress: true on a deploy, so a second push interrupts the first mid-copy.
• Running rsync --delete at production without testing the exclude list.
• Treating “the files copied” as a successful deploy, with no health check.
• Deploying straight from branch pushes with no approval step.
• Relying on log masking to keep a leaked secret out of the output.
• Never rotating the deploy key, including after someone leaves the team.

Best practices

• Dedicated deploy user, dedicated key, used for nothing else.
• Restrict the key in authorized_keys with restrict and a forced command.
• Scope sudo to the exact commands the deploy needs.
• Pin every action to a full commit SHA and let Dependabot bump them.
• Keep the deploy job small: fewer steps means fewer things that can read the secret.
• Set permissions: explicitly, defaulting to read-only.
• Verify host keys from a stored known_hosts value.
• Use environments with required reviewers and deployment rules for production.
• Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.
• End every deploy with a real HTTP request against the public URL.
• Rotate the deploy key on a schedule and whenever someone leaves.

FAQ

Is it safe to store an SSH private key in GitHub secrets?

Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.

Should I use a ready-made SSH deploy action?

They work and they are convenient. My preference is plain ssh and rsync in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.

Can I use OIDC instead of a stored key?

Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub’s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.

Should I allowlist GitHub’s IP ranges on my firewall?

Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer’s runners share those ranges. Narrowing what the key can do achieves more.

Self-hosted runner or hosted runner?

Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.

How do I roll back a bad deploy?

Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.

How often should the deploy key be rotated?

On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to authorized_keys and updating the secret, so there is no good reason for it to be a project.

The one thing to remember

The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.

So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger’s code did with your credentials.

Want this built or reviewed?

Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:

• Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.
• Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.
• Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.
• Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.
• Pinning and auditing third-party actions, plus repository settings that limit what can run.
• Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.

Send me your workflow file and the output of ssh deploy@yourhost 'id', and I will tell you what a stolen key would get.

The post Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully appeared first on John Nessime.

You go back through the logs and there it is, four hundred lines up, in a step that reported success: a download that returned an HTML error page, got piped into tar, and produced an archive containing nothing useful. The shell reported the exit status of tar, not of curl. The build carried on. The pipeline was never lying on purpose. It just never asked the right question.

Two things go wrong with pipelines, and only one of them gets talked about. The famous one is slow feedback: thirty-four minutes of build and integration tests before a linter tells you a file is missing a trailing newline. The expensive one is a pipeline that passes when it should have stopped. A fail fast pipeline fixes the first problem. Making it explain itself fixes the second, and that is the half most teams skip.

This post covers both: how to order stages so real failures surface in seconds, how to stop the shell from hiding non-zero exits, how to design failure output that hands the next engineer the fix, and the troubleshooting steps for when a pipeline stubbornly refuses to fail at the right moment.

The failure that actually bites: a pipeline that lies

Start here, because everything else is optimisation on top. A slow pipeline wastes your afternoon. A silently passing pipeline ships a broken artifact and burns a day of debugging in production, usually while somebody is on a call with a customer.

The shell is where most of this happens. Bash gives you three defaults that are wrong for CI, and one line fixes all three.

#!/usr/bin/env bash
set -euo pipefail

# Without pipefail this line exits 0 even when curl fails,
# because the shell reports the status of the LAST command
# in the pipe, which is tar.
curl -sSL https://example.com/artifact.tar.gz | tar -xz

What each flag actually protects

• -e stops the script when a command exits non-zero. Without it, a failed database migration is followed by a successful echo, and the script exits 0 because the last command succeeded.
• -u turns an unset variable into an error. This is the one that saves you from rm -rf "$PREFIX/build" expanding to rm -rf "/build" when someone renames an environment variable in the pipeline settings and forgets the script.
• -o pipefail makes a pipeline return the first non-zero status instead of the last one. Every curl | tar, cat | jq, and docker logs | grep in your scripts is currently reporting only the exit code of the thing on the right.

The exit status is only half the story though. Plenty of tools return 0 while doing something useless, and curl is the classic offender.

# curl exits 0 here even when the server returns 500,
# and you end up untarring an HTML error page.
curl -sSL "$ARTIFACT_URL" | tar -xz

# --fail          makes curl exit non-zero on 4xx and 5xx
# --show-error    keeps the reason visible even with --silent
# --location      follows redirects instead of saving a 302 body
curl --fail --silent --show-error --location "$ARTIFACT_URL" | tar -xz

Now the download failing stops the build, and pipefail makes sure that failure propagates past the pipe. The first fixes the check, the second makes sure the check is heard.

Where set -e quietly gives up

This trips people up constantly. set -e is suspended whenever a command’s exit status is being consumed by something else. It is not a safety net, it is a default.

set -e

# -e does NOT stop the script here.
# The if statement consumes the exit status.
if grep -q "TODO" src/main.go; then
  echo "found"
fi

# -e does NOT stop the script here either.
# The left side of || is explicitly allowed to fail.
run_migrations || echo "migrations failed, continuing anyway"

# -e DOES stop the script here.
run_migrations

The same trap applies inside containers. On Debian-based images /bin/sh is dash, which has no pipefail option at all, so putting set -o pipefail inside a RUN line fails outright. Set the shell at the top of the Dockerfile instead. Hadolint flags this as DL4006 and it is worth wiring into your static stage.

# Set this before any RUN instruction that contains a pipe.
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl --fail --silent --show-error --location 
      https://example.com/tool.tar.gz | tar -xz -C /usr/local/bin

Order stages by cost, not by category

Most pipelines are laid out the way the team thinks about work: tests, then build, then security, then deploy. That grouping is comfortable and it is the wrong axis. Order by two things instead, cost and failure probability.

The rule I use: every stage should be cheaper than the one after it and more likely to fail than the one after it. If a check catches ten percent of your broken commits and takes fifteen seconds, it goes in front of the check that catches two percent and takes nine minutes. That is the whole design principle behind a fail fast pipeline.

1. Syntax and static analysis. shellcheck, hadolint, yamllint, terraform validate, tflint, formatter checks, type checks. Seconds, no dependencies, no network. This stage catches an embarrassing share of broken pushes.
2. Unit tests. No database, no containers, no network. If they need any of those, they are integration tests and belong further down.
3. Secrets and dependency scanning. Fast, and a leaked credential is worth stopping before you build an image around it.
4. Build. This is where you start spending real minutes and cache space.
5. Integration and end-to-end. Slow, flaky, expensive to run. Everything above should have already caught the boring failures.
6. Deploy and post-deploy verification.

A note on the trade-off, because this ordering costs something. Splitting the fast checks into their own stage adds a checkout and a container pull before your unit tests even start. On a busy shared runner that overhead is real. If your static stage takes twelve seconds of work and forty seconds of setup, merge it into the test job and run it as the first step instead. The principle is failure order, not job count.

The other thing that keeps the fast lane fast is caching, and it is the piece people get wrong in the opposite direction: they cache so aggressively that a stale dependency tree hides a genuine break. Cache the package manager’s download directory, not the resolved dependency tree, and key the cache on the lockfile hash.

Let the platform kill work you already know is doomed

Ordering stages is only useful if the platform actually stops when something breaks. Three settings do most of the work, and they exist under different names on every runner.

GitHub Actions

concurrency:
  # A new push to the same branch cancels the run that is
  # already in flight. Nobody needs results for a commit
  # that has been superseded.
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 5          # a hung job is worse than a failed one
    steps:
      - uses: actions/checkout@v4   # pin to a full commit SHA in production
      - run: shellcheck scripts/*.sh
      - run: hadolint Dockerfile

  test:
    needs: static               # will not start until static passes
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false          # see note below
      matrix:
        suite: [unit, contract]
    steps:
      - uses: actions/checkout@v4
      - run: make test-${{ matrix.suite }}

That fail-fast: false looks like it contradicts the whole post, so it is worth explaining. Matrix fail-fast defaults to true: one leg fails, the rest get cancelled. That is right when the legs are redundant and you only need to know that something broke. It is wrong when the legs tell you different things. If your matrix is three runtime versions and only one fails, cancelling the others hides whether this is a version-specific problem or a general one, and you will burn a rerun finding out. Fail fast on the pipeline, not on the diagnosis.

timeout-minutes is the setting people skip and regret. A job waiting on an interactive prompt, a dead package mirror, or a container that never becomes healthy will sit there consuming a runner until someone notices. Set a timeout on every job, generous enough that a slow day does not trip it and tight enough that a hang gets caught inside one coffee break.

GitLab CI

stages: [static, test, build, deploy]

default:
  interruptible: true     # allows auto-cancel when a newer pipeline starts
  retry:
    max: 2                # capped at 2 by GitLab, so 3 attempts total
    when:
      # Retry only on infrastructure problems.
      # Never retry script_failure: that is your code failing,
      # and retrying it just hides a flaky test.
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

lint:
  stage: static
  timeout: 5m
  script:
    - shellcheck scripts/*.sh

integration:
  stage: test
  needs: [lint]           # DAG: starts as soon as lint passes
  script:
    - make integration

The retry: when list is the important bit and it is the one people get lazy about. A bare retry: 2 retries on everything including a genuine test failure, which converts an intermittent bug into an invisible one. Scope retries to the failure classes that are genuinely the infrastructure’s fault. Jenkins has failFast true for parallel stages, Buildkite and CircleCI have their own equivalents, and the reasoning transfers cleanly.

A failure should hand you the answer, not the search

Here is the test I apply to a pipeline failure. Can someone who did not write the change, at an inconvenient hour, fix it from the failure output alone without opening the source? If not, the failure message is incomplete.

Four things need to be in there:

• What failed, named precisely. Not “build failed” but the command and its exit code.
• Why, in terms of the input rather than the internals. “lockfile is out of date with package.json”, not “npm ci exited 1”.
• Where, meaning the file, the line, or the config key that caused it.
• What to do next. One concrete action, even if it is just a runbook link.

In shell, a small helper plus an ERR trap gets you most of the way there.

#!/usr/bin/env bash
set -euo pipefail

fail() {
  echo "ERROR: $1" >&2
  echo "  fix: $2" >&2
  exit 1
}

# Catches anything you did not write an explicit check for,
# and tells you the line and the command that blew up.
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

[[ -f config/app.yaml ]] || fail 
  "config/app.yaml is missing" 
  "copy config/app.yaml.example and set DATABASE_URL"

[[ -n "${IMAGE_TAG:-}" ]] || fail 
  "IMAGE_TAG is empty" 
  "check the tag job output; it sets IMAGE_TAG from the git SHA"

Note ${IMAGE_TAG:-} rather than $IMAGE_TAG. Under set -u the bare form aborts with a shell error before your nice message ever prints, which is technically correct and practically useless.

On GitHub Actions you can go one step further and attach the error to the file and line, so it shows up as an annotation on the pull request instead of something to be dug out of a log.

# Renders as a red annotation on that exact line in the diff view.
echo "::error file=deploy/values.yaml,line=42::image tag is empty"

# Visible, but does not fail the job.
echo "::warning file=Dockerfile,line=7::apt cache not cleaned"

Two more habits worth building. Upload logs, test reports, and screenshots as artifacts on failure, conditionally, so the evidence survives the job that produced it. And emit test results as JUnit XML so the platform renders a list of failed test names instead of asking people to scroll. Both are cheap and both cut the time between “it broke” and “I know why” more than any clever retry logic will.

Exit codes are a contract, so use more than one

Most scripts have two states: 0 and “something went wrong”. That collapses distinctions the pipeline needs. Terraform gets this right and it is a good model to copy.

# -detailed-exitcode changes the meaning of the exit code:
#   0 = succeeded, no changes
#   1 = error
#   2 = succeeded, changes present
set +e
terraform plan -detailed-exitcode -out=tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift. Skipping apply." ;;
  2) echo "Changes detected. Holding for approval." ;;
  *) echo "Plan failed. See log above." >&2; exit "$code" ;;
esac

The set +e around it is deliberate: with -e active, exit code 2 would kill the script before you got to interpret it. This is one of the few places where turning the flag off is the correct move, and it is worth a comment in the file explaining why so nobody “fixes” it later.

Do the same in your own scripts. Reserve a code for “the thing you asked for is not possible” versus “I could not reach the API” versus “your input is wrong”. The pipeline can then retry one and refuse to retry the others, which is the difference between a retry that absorbs a runner hiccup and a retry that hides a race condition for six months.

Gate on what you care about, not what is easy to check

This is the deploy-stage version of the same lie. Plenty of commands confirm that a request was accepted, not that the thing you wanted happened.

# Returns 0 the moment the API server accepts the manifest,
# even if every pod goes into CrashLoopBackOff a second later.
kubectl apply -f k8s/

# This is the step that actually waits for the rollout,
# and fails the job if it does not converge.
kubectl rollout status deployment/api --timeout=180s

# And this proves the application answers a request,
# not just that a pod reports Ready.
curl --fail --silent --show-error https://api.example.com/healthz

Run that last check against the public hostname, through your CDN, rather than against the pod directly. If you front the app with Cloudflare or similar, a smoke test that skips the edge will happily pass while real users hit a cached error page or a misrouted origin. Same reasoning applies to the origin host itself: if the app runs on a VPS from a provider like InterServer behind a reverse proxy, verify through the proxy, because that is the path traffic takes.

Then track how long each stage takes over time. Pipeline duration and failure rate per stage are perfectly ordinary metrics to ship into Prometheus and graph in Grafana, or into whatever platform you already use for application monitoring. When the fast lane creeps from forty seconds to four minutes, that is a slow regression nobody will notice from a single run.

Troubleshooting

The job passed but the artifact is wrong

Look for pipes first. Grep the scripts for | and check whether pipefail is set in that shell. Then check for tools that return 0 on failure: curl without --fail, docker run where the entrypoint swallows errors, anything wrapped in || true that somebody added during an incident and never removed.

set -e is set but the script keeps going

The failing command is almost certainly inside a condition, on the left of || or &&, or in a function that is itself being called from an if. Add set -x temporarily and read the trace to see exactly where control flow continued. Then use an explicit check with your fail helper rather than relying on -e at that point.

Works locally, fails in CI (or the reverse)

Usually a shell difference. Your terminal runs bash or zsh, the runner may execute the script with sh. Add an explicit shebang, and on GitHub Actions set the shell on the step or at the job level rather than assuming. Inside containers, remember dash has no pipefail and no [[ ]].

The job hangs until the platform kills it

Something is waiting on stdin. Package managers prompting for confirmation, ssh asking about a host key, a database client waiting for a password. Add the non-interactive flags, set CI=true where tools respect it, and put a timeout on the job so the next occurrence fails in ten minutes instead of sixty.

Fails on the first run, passes on rerun

Resist the retry. Log the failure with enough context to identify it later, quarantine the test into a non-blocking job if it is genuinely blocking the team, and give it an owner and a date. A blanket retry turns a flaky test into a permanent tax on everyone’s confidence in the pipeline.

The failure output is a wall of noise

Suppress progress bars and spinners in CI, they render as thousands of unreadable lines. Then print a short summary block at the very end of the job, after the noise, listing what failed and what to do. People read the bottom of a log first.

Common mistakes

• Writing set -e without -o pipefail and assuming the script is covered.
• Using || true to silence a step that fails intermittently, instead of finding out why.
• Running the slowest, flakiest checks before the ones that take fifteen seconds.
• Retrying on every failure class rather than only infrastructure faults.
• No job timeouts, so a hang consumes a runner until someone notices.
• Treating kubectl apply or a deploy API call as proof the deploy worked.
• Failure messages that name the internal function that threw, and nothing the reader can act on.
• Caching so aggressively that a stale dependency tree hides a real break.
• Discarding logs and test reports when the job fails, which is exactly when you need them.
• Turning off matrix fail-fast everywhere by reflex, so every broken commit costs the full matrix.

Best practices

• Put set -euo pipefail at the top of every shell script the pipeline runs, and set the Dockerfile SHELL before any piped RUN.
• Order stages so each one is cheaper and more likely to fail than the next.
• Give every job a timeout, and cancel superseded runs on the same branch.
• Use explicit checks with actionable messages instead of relying on -e to catch everything.
• Distinguish exit codes so the pipeline can react differently to different failures.
• Verify deploys with a rollout status check plus a real HTTP request through the public path.
• Upload artifacts and structured test reports on failure.
• Pin action and image references to immutable digests so a green pipeline stays reproducible.
• Track stage duration and failure rate over time, the same way you track application metrics.
• Fix the failure message the first time somebody has to ask what it means.

FAQ

What does “fail fast” actually mean in a CI/CD pipeline?

Two things. First, the pipeline stops as soon as it knows the result will be a failure, rather than finishing work whose outcome no longer matters. Second, the cheapest checks that are most likely to catch a problem run first, so the average failure surfaces in seconds. The second half is a design choice about ordering, not a platform setting.

Is set -e enough on its own?

No. It is suspended inside conditions, on the left side of && and ||, and after !. It also does nothing about pipes, which is why -o pipefail matters, or about commands that return 0 while failing, which is why explicit checks matter. Use it as a baseline, not as your only line of defence.

Should I disable matrix fail-fast?

Only when the legs give you different diagnostic information. If the matrix is runtime versions or operating systems and you need to know whether the break is universal or specific, let all legs run. If the legs are interchangeable shards of the same test suite, leave fail-fast on and save the compute.

How do I stop a pipeline from passing when a step really failed?

Work through three things in order. Set pipefail so pipes propagate failure. Add the flags that make tools return non-zero on failure, such as curl --fail. Then replace implicit trust with explicit assertions: after a deploy, check that the rollout converged and that the service answers a request. Most silent passes come from one of those three gaps.

Are automatic retries a good idea?

Only for failures that are the infrastructure’s fault: runner crashes, image pull errors, API timeouts from the CI platform itself. Never retry a script failure by default. And make retries visible, because a job that quietly succeeds on the third attempt is telling you something important that nobody sees.

How long should a fast feedback stage take?

Short enough that an engineer waits for it rather than switching context. Somewhere under a couple of minutes for the combined lint, format, and unit stage is a reasonable target for most codebases. Past that, people push and walk away, and you lose the benefit of fast feedback even though the pipeline is technically quick.

Does this apply to self-hosted runners too?

All of it, and timeouts matter more. On hosted runners a stuck job eventually hits the platform’s own ceiling. On your own machine it holds a slot until you intervene, so a hung job blocks the queue for everyone. Set explicit timeouts and make sure the runner cleans up its workspace between jobs.

The one thing to take away

Speed is the part everyone optimises and honesty is the part that actually costs money. A fail fast pipeline that finishes in ninety seconds is worth very little if one of those seconds contains a swallowed exit code. Fix the lying first: pipefail, tools that return non-zero when they fail, and assertions that check the outcome rather than the request. Then reorder for speed.

And judge every failure by one question. Could someone else fix this from the output alone, without reading the source? If the answer is no, the pipeline has not finished its job, no matter how quickly it went red.

Need help fixing a pipeline that lies to you?

If any of this sounds like your CI, this is the kind of work I take on. Typical engagements look like:

• Auditing an existing GitHub Actions, GitLab CI, or Jenkins pipeline for silently swallowed failures and false-green stages.
• Restructuring stage order and caching so the common failure surfaces in seconds instead of after the build.
• Hardening deploy verification: rollout gates, smoke tests through the real public path, and automatic rollback on a failed health check.
• Rewriting pipeline shell scripts with proper error handling, exit code contracts, and failure messages an on-call engineer can act on.
• Wiring pipeline duration and failure metrics into Prometheus and Grafana so regressions get caught early.

Send me the workflow file and a link to a failing run, and I will tell you what I would change first.

The post Green Check, Broken Build: How to Build a Fail Fast Pipeline That Explains Why appeared first on John Nessime.