That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That gap is where the real cost lives, and it is almost never on anyone’s dashboard. The build duration graph looks fine. The team still says shipping is slow. Somebody proposes buying bigger runners, and six weeks later the graph has moved a little and the complaints are identical.

Most CI/CD pipeline bottlenecks are not the thing people go looking for. Compute is usually the smallest slice. The expensive parts are minutes spent waiting for a runner, a cache that reports a hit and delivers nothing, a stage blocking four jobs that do not depend on it, and a required check sitting at “Pending” until someone notices. This post is organized by those failure families rather than by tool. Examples use GitHub Actions and GitLab CI, but the shapes apply to Jenkins, Buildkite, CircleCI and Argo Workflows too.

First, split the number you are staring at

“The build takes eleven minutes” is not a measurement. It is three measurements added together and then thrown away:

• Queue time. From job creation to a runner picking it up. You pay this in wall clock, not compute.
• Compute time. The job actually running. The only part most dashboards show.
• Gate time. Waiting on a human, an approval, a shared environment, or a check that never fired.

Until you split those, every optimization is a guess. On GitHub Actions the jobs API hands you the split directly, because each job carries a creation timestamp and a start timestamp:

gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs 
  --jq '.jobs[] | {name, created_at, started_at, completed_at}'

Creation to start is queue. Start to completion is compute. Run that across your last fifty runs on the default branch and the answer is usually sitting there in plain sight. GitLab reports the same idea on its jobs API, with a queued duration alongside the run duration. If the two numbers are roughly equal, you have a capacity problem rather than a build problem, and faster CPUs will fix half of nothing.

Queue time: the minutes you pay for twice

Queue time has two causes and they need different fixes.

Not enough runners

On hosted runners you are bounded by the concurrency your plan allows. On self-hosted you are bounded by whatever fleet you provisioned, which for most small teams is a fixed number of always-on boxes that sit idle overnight and saturate at 4pm.

If you run your own fleet on Kubernetes, Actions Runner Controller is where most people land. It creates runner scale sets that scale on workflow demand and can drop to zero when the queue is empty, with ephemeral pods so every job starts clean. That last part matters more than the scaling: a persistent runner accumulates state, and state is where “works on my machine” comes back as a CI flake.

If Kubernetes is more machinery than you want, a couple of dedicated boxes from a provider like Contabo or InterServer running the runner agent under systemd gets you a long way, with hosted runners as overflow when the queue backs up. Managed runner services such as Depot and BuildJet sell faster hardware behind the same interface. Their honest pitch is that you stop maintaining a fleet. The honest counterpoint is another vendor in the path of every merge, and a bill that scales with commit volume.

Runners busy on work that is already obsolete

Check this one first because it is free. Push three commits to a branch in five minutes and by default you get three pipelines. The first two are answering a question nobody is asking anymore, and they are holding the runners the third one needs.

On GitHub Actions, a concurrency group keyed on the ref does it:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Put that at workflow level on PR builds. Do not put it on a deploy workflow with cancellation enabled, or you will eventually kill a release halfway through and spend an afternoon working out what actually shipped.

GitLab splits the same job across two settings that have to agree. The interruptible keyword marks a job as safe to kill and defaults to false. The workflow-level auto-cancel setting decides what happens on a new commit:

workflow:
  auto_cancel:
    on_new_commit: interruptible

test:
  interruptible: true
  script:
    - make test

deploy:
  interruptible: false
  script:
    - make deploy

The default mode is conservative, which means once any non-interruptible job has started, the whole pipeline stops being cancelable. Setting it to interruptible cancels only the jobs that opted in and leaves the rest running. That distinction is the difference between a policy that works and one that quietly does nothing, and it is the usual reason teams turn auto-cancel on and see no change.

Caches that look like they are working

My favorite family of CI/CD pipeline bottlenecks, because the failure is completely silent. The step is green. The log says the cache was restored. The install takes exactly as long as before.

Branch scoping

GitHub Actions caches are scoped to the branch that created them, plus the default branch as a fallback. A feature branch reads its own caches and the default branch’s caches. It cannot read a sibling branch’s cache.

The consequence catches a lot of people: if your caching step only runs on pull requests, the default branch never populates a cache, so every new branch starts cold forever. Make sure at least one workflow on the default branch writes it. Runs triggered from fork pull requests get read-only cache access, so those jobs restore but never save.

Keys that never hit, or hit and never update

Cache entries are immutable. Once a key exists, writing to it again does nothing. So a key like node-modules-v1 gets populated once, on some machine, in some state, and then freezes. A key that hashes the lockfile is correct but creates a brand new entry on every dependency change, which burns through storage.

The pattern that survives contact is an exact key plus a prefix fallback: hash the lockfile for the primary key, then use restore keys to fall back to the most recent entry sharing that prefix. You get a partial restore and a fast install instead of a cold one.

Then watch the ceiling. Repositories have a default cache storage limit, entries untouched for a week are removed, and once you hit the limit the least recently accessed entries get evicted. Push enough large caches and you get thrashing, where entries are created and evicted at roughly the rate you create them. The tell is a hit rate that looks random. Check the repository’s cache usage in Actions settings before assuming the config is wrong.

Docker layer cache that does not cross machines

On ephemeral runners the local BuildKit cache dies with the machine, so it has to be exported somewhere. Registry-backed cache is the one I reach for first:

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:${GIT_SHA} 
  --push .

Two details do the work. mode=max exports intermediate stage layers, not just the ones that landed in the final image. If the expensive work happens in a builder stage that gets discarded, the default mode caches none of it, which is exactly the case you were trying to fix. Inline cache, the kind that ships alongside the image, only supports the minimal mode, so it cannot help multi-stage builds either.

The second detail is a trap. BuildKit cache mounts, the RUN --mount=type=cache kind, are not part of the exported layer cache. They live on the builder. Restoring a registry cache does not restore them, so if the instruction runs again the package manager starts from an empty download cache:

# syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

That line is excellent on a persistent builder and close to useless on a runner destroyed after every job. Know which one you have before you credit it with a speedup.

Pipelines shaped like a staircase

Stages are a scheduling constraint most pipelines inherit by accident. In a stage-based model nothing in the test stage starts until everything in the build stage finishes. Put a forty-second lint job in the same stage as a six-minute compile and your lint feedback arrives six minutes late for no reason.

GitLab’s fix is needs, which lets a job start as soon as its own dependencies finish and ignore stage order entirely:

lint:
  stage: test
  needs: []
  script:
    - make lint

unit:
  stage: test
  needs: ["build"]
  script:
    - make unit

needs: [] means start immediately, wait for nothing. Lint now reports in under a minute while the compile is still running.

GitHub Actions has the opposite default: jobs run in parallel unless you add needs. So the equivalent problem there is over-declared dependencies, usually added early to force a readable graph and never revisited. Go read your needs lines and ask which are real data dependencies and which are narrative ordering.

The trade-off is real. A wide graph surfaces every failure at once and costs more compute per run. A narrow graph fails fast and saves minutes, but developers make several round trips to find three problems. For PR pipelines, wide is usually right. For expensive nightly jobs, narrow is.

The test suite is not one bottleneck

“Tests are slow” gets treated as a single problem. It is usually three, and they want different fixes.

• The suite is serial. Split by shard across parallel jobs. The cost is per-job setup overhead paid N times, so past a point more shards makes things slower. Find your own break-even instead of assuming more is better.
• The suite is flaky. A retry costs more than one rerun. It costs attention, a context switch, and eventually the team’s belief that red means anything. Quarantine flaky tests into a non-blocking job with an owner and a deadline. Leaving them in the required path while everyone knows to just re-run is worse than deleting them.
• The suite tests the wrong layer. A test that boots a database and a message broker to assert a validation rule is a unit test wearing a costume. Slowest fix, largest payoff.

Before touching any of it, get per-test timing out of your test runner and sort descending. Most suites have a handful of tests that dominate, and they are rarely the ones people suspect.

The wait that nobody is measuring

Here is the one that costs days rather than minutes, and it appears in no CI metric because from CI’s point of view nothing is happening.

If a workflow is skipped because of path filtering, branch filtering, or a skip directive in the commit message, its checks stay pending. Not skipped. Pending. And if that check is required, the pull request cannot merge. Nothing fails, nothing alerts, the PR just sits there looking almost ready until someone investigates by hand.

This bites hardest in monorepos, which is exactly where path filters are most attractive. You add paths to skip the service test suite on documentation-only changes, you save real minutes, and you accidentally build a merge trap.

The distinction that matters: a job skipped by an if condition inside a workflow that did run reports success. A workflow that never ran at all reports pending. So either keep the workflow triggering on everything and move the filtering inside, or pair each filtered workflow with a companion using the same name and inverted path filters that does nothing and passes. GitHub documents that second pattern and it works, but it means two files that must stay in sync.

The other gate-time offenders are less subtle and just as expensive: one shared staging environment serializing every team, and approvals routed to a single person asleep in another time zone. Plot merge timestamps against last-commit timestamps for a month. The shape of that distribution tells you which problem you have.

Troubleshooting: finding your bottleneck in an afternoon

1. Pull the last fifty runs on your default branch and compute queue time and compute time separately. If queue is a meaningful share, fix capacity and cancellation before anything else.
2. Read your slowest run as a graph, not a list. Wide gaps between a dependency finishing and its dependent starting are either queue or bad ordering.
3. Compare install duration on a cache hit against a cache miss. If they are similar, the cache is decorative.
4. Check repository cache storage usage. Thrashing looks like an erratic hit rate with no config changes behind it.
5. Count retries. The rate of second attempts on the same commit is your flakiness tax, and it is usually higher than anyone guesses.
6. Measure merge latency, last commit to merge, across a month. The difference between that and pipeline duration is gate time, and that is where the days are.

Pipeline run data is worth shipping somewhere queryable. Most CI platforms emit webhooks on job completion; feeding those into Prometheus and Grafana, or a hosted platform like Grafana Cloud or Datadog if you would rather not run the storage, turns “the build feels slow lately” into a graph you can argue with.

Common mistakes

• Buying bigger runners before splitting queue time from compute time. The most common wasted spend in CI.
• Turning on auto-cancel without setting the per-job interruptible flags, so the setting is technically on and functionally off.
• Caching a directory the tool does not read from, then declaring victory because the step went green.
• Using the default Docker cache export mode on a multi-stage build and wondering why the builder stage rebuilds every time.
• Adding path filters to workflows that back required checks, creating pull requests that can never merge.
• Optimizing the pipeline that runs on merge instead of the PR pipeline developers actually wait on.

Practices that hold up

• Treat queue time as a first-class metric with its own threshold, not a footnote on build duration.
• Cancel superseded runs on branches, never on the default branch and never on deploys.
• Make the default branch populate every cache other branches depend on.
• Order jobs by how fast they fail, not by how the file reads top to bottom. Cheap checks first, always unblocked.
• Give every flaky test an owner and an expiry date, or take it out of the blocking path.
• Re-measure after every change and keep the number. Pipelines rot, and the bottleneck moves as soon as you fix the current one.

Frequently asked questions

How do I find the bottleneck in my CI/CD pipeline?

Split every job into queue time and compute time using the timestamps your platform’s API already exposes, then compare pipeline duration against real merge latency. Those two comparisons put you in one of three buckets: capacity, build work, or gates. Fixing the wrong bucket is why most optimization efforts show no result.

Are self-hosted runners faster than hosted ones?

Per job, often yes, because you pick the hardware and can keep caches warm. Across a working day, frequently no, because a fixed fleet queues under load while a hosted pool does not. The right answer depends on which number is hurting you, compute or queue, which is why measurement comes first. A hybrid setup, self-hosted by default with hosted overflow, avoids committing to either.

Why is my cache not speeding anything up?

Usual suspects in order of frequency: the cache is scoped to a branch that never populated it, the key is static so it froze at whatever went in first, the cached directory is not the one the tool reads, or entries are evicted before the next run. Compare install duration on a hit against a miss. Identical duration means the restore is doing nothing regardless of what the log says.

Should I parallelize tests or make them faster?

Parallelize first because it is cheap and reversible, then fix the slow tests, because parallelism has a floor. Every shard pays setup cost, so a suite split too many ways spends more time booting containers than running assertions. Sharding buys time to do the real work. It is not the real work.

Why does my pull request say pending forever?

Almost certainly a required status check attached to a workflow that was skipped by path filtering, branch filtering, or a commit message directive. A skipped workflow reports pending rather than skipped, so the merge requirement is never satisfied. Either drop the path filter and skip inside the workflow instead, or add a companion workflow with the same name that always passes.

Do bigger runners actually help?

They help work that is genuinely CPU or memory bound, such as compilation and container builds. They do nothing for queue time, network-bound dependency downloads, tests waiting on external services, or gate time. Before upgrading, confirm the slow job is saturating the resource you are about to buy more of.

The one thing worth remembering

Most CI/CD pipeline bottlenecks are waiting, not working. Queue time, cache misses that report as hits, jobs blocked behind stages they do not depend on, and checks that sit pending because a workflow was never triggered. None of them show up on a build duration graph, and all of them are cheap to fix once you can see them. Split the number before you spend anything; the measurement takes an afternoon and usually points at a config change rather than a purchase order.

Need someone to find where your pipeline is losing time?

This is a large part of what I do as a freelance DevOps engineer. Typical work looks like:

• Instrumenting your pipeline so queue time, compute time and merge latency become three separate numbers you can watch over time
• Auditing GitHub Actions and GitLab CI caching to find the restores that report success and deliver nothing
• Restructuring stage-based pipelines into dependency graphs so cheap checks report first
• Setting up autoscaling self-hosted runners, on Kubernetes with Actions Runner Controller or on plain VMs, with hosted overflow
• Fixing Docker build caching for multi-stage images on ephemeral runners, including registry-backed export
• Untangling required checks and path filters in monorepos so pull requests stop getting stuck

Send me a workflow file, a job timeline, or the output of your slowest run, and I will tell you where the minutes are going before we talk about anything else.

The post CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go appeared first on John Nessime.

Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource’s name inside the module. The author had tidied up a resource "aws_db_instance" "this" into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.

That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.

This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.

The bill for a module arrives at upgrade time

Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: module.database.aws_db_instance.this. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.

For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.

The fix has existed for a long time and it is genuinely good: the moved block. A module author who renames something declares the rename in code, ships it with the release, and every consumer’s next plan updates state instead of destroying infrastructure.

# Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}

The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its moved blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.

So here is the whole thesis, and everything below is elaboration: a module is worth reusing if the next version bump is cheap. Everything else is secondary.

Five signals that reusable Terraform modules are worth adopting

These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.

1. It keeps state addresses stable, or tells you when it does not

Open the repository. Search for moved. Read the changelog for the last major release and see whether renames are called out.

A module with no moved blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says “v3.0 renames the launch template resource, here are the moved blocks” is telling you it has thought about consumers.

2. The input surface is narrow and typed

Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.

Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.

variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}

Three things are doing work there. optional() with a default means a caller who wants the simple case writes two lines instead of six. nullable = false means a caller cannot pass null and get a confusing failure three resources deep. The validation block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.

That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.

3. You can pin it, and the pin means something

This trips people up because the two common source types behave differently.

# Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}

The ~> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as ~> 5.0.1 it accepts patches only. Which one you want depends entirely on whether you trust the author’s reading of semantic versioning, and the honest answer for third-party modules is usually “not enough to auto-accept minors in production”.

For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at init time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.

4. It has tests you can run without a cloud account

Terraform’s built-in test framework uses .tftest.hcl files and the same HCL you already write. The critical detail for module authors is that a run block set to command = plan creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.

# tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}

Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.

OpenTofu runs the same format through tofu test, and additionally recognises a .tofutest.hcl extension for tests that should only run there. Useful if you maintain modules that target both tools.

5. Somebody answers issues

Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.

Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when init refuses to resolve a version constraint.

Three kinds of module, and which ones actually pay off

Not everything that can be a module should be. It helps to name the three shapes you see in the wild.

The thin wrapper

One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.

There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.

The composition module

Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.

Most modules worth reusing live here.

The landing zone

An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.

Public modules or your own?

The case for public modules is stronger than the “we have specific needs” crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.

The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.

My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.

There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp’s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.

For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.

Making your own modules worth reusing

If you are on the authoring side, this is the order I would do things in.

1. Write it inline first, twice. Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module’s interface, discovered rather than guessed.
2. Extract with moved blocks in the same commit. Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.
3. Type the inputs properly. Objects with optional() defaults over flat strings, nullable = false where null makes no sense, validation blocks with error messages that name the fix.
4. Prefer for_each over count for anything collection-shaped. With count, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.
5. Generate the docs. terraform-docs reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.
6. Lint and scan on every PR. terraform fmt -check, terraform validate, tflint for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.
7. Tag releases, and mean it. A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.

One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.

Troubleshooting the plans that scare you

“Forces replacement” after a version bump

Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.

terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'

If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version’s source. A rename with no moved block is the usual answer, and you can write the moved block yourself in your root configuration as a workaround.

A module with a provider block inside it

Modules should declare which providers they need, not configure them. A module that contains a configured provider block rather than just a required_providers entry limits what callers can do with it, including using for_each or count on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.

Changes to the module source are not picked up

Remote modules are downloaded into the working directory during init. Changing a ref or a version constraint in your configuration does not re-download anything on its own. Run terraform init -upgrade. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.

Registry version list looks stale

A registry’s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry’s view after any change to that integration.

Common mistakes

• Pinning to a branch. ?ref=main is not a pin. It is an unannounced deployment scheduled for whenever someone next runs init.
• Modularising after one use. You end up encoding the first use case’s assumptions as the abstraction, then bolting flags onto it for every case after.
• Copying environments instead of parameterising them. Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.
• Passing every provider argument straight through. If a module’s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.
• Deleting moved blocks too early. They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.
• Applying a plan you did not read. Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.

How I decide whether to adopt a module

Roughly ten minutes, in this order:

1. Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.
2. Grep for moved. Its presence tells me the author thinks about consumers.
3. Read the last major release notes. Are breaking changes explained, with an upgrade path?
4. Check for a tests directory and whether CI runs on pull requests.
5. Look at the issue tracker’s recent activity, not its total count.
6. Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.

And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.

Frequently asked questions

How many input variables is too many for a Terraform module?

There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.

Should I use public modules or write my own?

Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else’s.

Do Terraform modules work with OpenTofu?

Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.

How do I pin a module version from a Git source?

The version argument only applies to registry sources. For Git, append a ref query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at init time, which defeats the purpose of pinning entirely.

What happens if a module removes a resource in a new version?

Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.

Can I test a Terraform module without cloud credentials?

Yes, for a useful subset. Run blocks set to command = plan evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.

Do I need a private registry to share modules internally?

No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.

Conclusion

Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.

The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.

Need a second pair of eyes on your module library?

I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:

• Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations
• Adding moved blocks and upgrade notes to in-house modules so version bumps stop being incidents
• Narrowing wide module interfaces with typed objects, optional() defaults and validation rules that fail with useful messages
• Setting up module CI: fmt, validate, tflint, policy scanning and generated docs on every pull request
• Writing terraform test suites that run on plan, including negative tests that prove your validation rules actually fire
• Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all

Send me a module’s variables.tf and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.

The post Terraform Modules Worth Reusing (And the Ones That Cost You) appeared first on John Nessime.

Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.

This post covers how to keep Kubernetes secrets out of Git in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.

The failure that actually bites: encrypted in Git, wide open in the cluster

Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run kubectl get secret -o yaml in that namespace reads the value in one pipe through base64 -d.

So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster’s own blast radius.

Three things need to be true alongside whichever tool you pick:

• Encryption at rest is on. A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an EncryptionConfiguration, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.
• RBAC on secrets is actually restrictive. A depressing number of clusters grant blanket get and list on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.
• Secrets are mounted as files, not shoved into environment variables where you can help it. Environment variables leak through /proc/<pid>/environ, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.

Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:

# Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default

If that returns yes, your Git hygiene is not the weakest link in the chain.

Already leaked? Rotate first, rewrite second, and understand what rewriting does not do

Order matters here more than technique. GitHub’s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker’s ability to use it. History surgery is the optional follow-up, not the fix.

The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues’ clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.

When you do rewrite, use git filter-repo. It has replaced filter-branch as the recommended tool and it is dramatically faster on any repo with real history.

# Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags

Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A git pull will happily drag the old objects back in.

Two families of solution, and how to tell which one you are in

Every approach to this problem is either encrypt the value and commit the ciphertext, or commit a pointer and fetch the value at runtime. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.

Sealed Secrets: the lowest-friction way to start

A controller in the cluster holds an RSA private key and publishes the matching certificate. The kubeseal CLI encrypts a Secret manifest against that certificate and produces a SealedSecret custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.

# Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  > pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='<value>' 
  --dry-run=client -o yaml > db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml < db-auth.yaml > db-auth-sealed.yaml

rm db-auth.yaml

Because you can hand out pub-cert.pem, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.

Two details that catch people. First, sealing is scoped by default: a SealedSecret is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with --scope namespace-wide or --scope cluster-wide if you genuinely need to. Second, the Helm chart names the controller sealed-secrets while the CLI looks for sealed-secrets-controller by default, so --controller-name is not optional in practice.

Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.

SOPS with age: encrypt files, not just Kubernetes objects

SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with age, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.

A .sops.yaml at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:

creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient

The encrypted_regex line is the one worth understanding rather than copying. Without it SOPS encrypts everything including metadata, and you lose the readable-diff property that made this approach attractive. Restricting it to data and stringData keeps names, namespaces and labels reviewable in a pull request.

Flux decrypts SOPS natively: you point a Kustomization at a Secret holding the age private key via decryption.provider: sops and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.

SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain .env. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.

External Secrets Operator: stop putting the value in Git at all

The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password

This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for refreshInterval, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.

The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.

One live upgrade trap. The external-secrets.io/v1beta1 API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to v1, which produced permanent drift in Argo CD for anyone whose Git manifests still said v1beta1. Argo sees v1 in the cluster, Git says v1beta1, and it reconciles forever. Update your manifests to v1 before you cross that boundary, not after.

Secrets Store CSI Driver: when a Secret object is one object too many

Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.

The key you forgot to back up

This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.

Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every SealedSecret in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.

• Back up the sealing key out of band. The controller’s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.
• Same for age. The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.
• Add a second recipient for anything that matters. SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.
• Know your re-key procedure before you need it. For SOPS that is sops updatekeys across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.

Stopping plaintext from reaching the repo in the first place

All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.

1. Put a scanner in the pre-commit hook. gitleaks and trufflehog both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.
2. Run the same scan in CI, on the whole history, not just the diff. The diff-only scan misses everything that predates the day you added the scanner.
3. Turn on server-side push protection. GitHub’s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone’s local setup.
4. Make the plaintext path short-lived by construction. Never write a plaintext manifest to a tracked path. Use /tmp, or pipe straight into the encrypt step so the value never lands on disk at all.
5. Add *.dec.yaml, secret.yaml and friends to .gitignore as a convention. Not a control, just one less way to be careless.
6. Remember your shell history. A --from-literal with a real password ends up in ~/.bash_history in cleartext. Use --from-file, or read from a variable, or prefix the command with a space if your shell is configured to skip those.

Troubleshooting the four things that go wrong

The SealedSecret applies cleanly but no Secret appears

Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.

# The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml

The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.

The ExternalSecret sits there doing nothing

The status conditions carry the real message, and the events under describe usually name the provider error directly.

kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend

Check the store before the secret. If the SecretStore is not Ready, nothing referencing it will ever sync, and the error you want is on the store, not the ExternalSecret. When using a ClusterSecretStore, remember that any secret reference inside it needs an explicit namespace, because a cluster-scoped object has no namespace of its own to fall back on.

Argo CD or Flux reports permanent drift

Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO v1beta1 removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.

SOPS decryption fails only in the cluster

You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in .sops.yaml but running sops --encrypt with an explicit --age flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file’s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.

Common mistakes

• Rewriting history before rotating the credential, and treating the rewrite as the remediation.
• Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.
• Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.
• Never backing up the sealing key or the age identity.
• Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.
• Encrypting the entire manifest with SOPS instead of just data and stringData, and losing every useful diff.
• Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.
• Piping a Helm release’s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.

How I would actually decide

Skip the feature matrix and answer three questions in order.

1. Do you already run a secret store? If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.
2. Do you have config outside Kubernetes? Terraform variables, Ansible vars, application .env files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.
3. Neither? Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.

Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.

FAQ

Is base64 in a Kubernetes Secret a form of encryption?

No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.

Should I use Sealed Secrets or External Secrets Operator?

If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.

Do I still need to rewrite Git history after rotating a leaked credential?

Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.

Can Argo CD decrypt SOPS files on its own?

Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.

Is it safe to commit the Sealed Secrets certificate or the age public key?

Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.

The one thing worth remembering

If you take one idea away, make it this: the work to keep Kubernetes secrets out of Git and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of SealedSecret resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.

Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for filter-repo, because rotation is the part that actually stops the bleeding.

Need help sorting out secrets in your cluster?

This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:

• Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output
• Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure
• Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover
• Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions
• Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces
• Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives

If you have a controller log, a stuck ExternalSecret, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.

The post Keep Kubernetes Secrets Out of Git Without Kidding Yourself appeared first on John Nessime.

Then it clicks. Somebody logged into wp-admin last month, saw an update notice, clicked it. The plugin got updated on the server and never in git. Last night’s deploy synced the repo over the top and put the old version back, along with the vulnerability the update was patching. No error, no warning. The deploy did exactly what it was told.

That is the failure mode that kills most attempts at Git-based WordPress deployment, and it has nothing to do with your pipeline. WordPress is designed to modify itself. It installs plugins, updates core, writes files, all from the browser. The moment you put it under version control you have two systems that both believe they own the file tree, and they will quietly overwrite each other until somebody notices.

This post covers the whole chain: what belongs in the repo, how to shut the second write path, how configuration differs per environment, which way the database is allowed to travel, and how to make staging safe enough that you can restore production data into it without emailing your customers by accident.

The two things that make WordPress awkward under Git

Everything in this workflow follows from two facts.

State is split across files and a database. Your theme is code. Your posts are data. But so are your plugin settings, your menus, your widget layouts, and your permalink structure, and those live in the database next to the content. Git only manages half your site, and it is not obvious which half a given change landed in.

The two halves travel in opposite directions. Code moves forward, local to staging to production. Content moves backward, production to staging to local. Once a site is live, production is the only authoritative source of orders, comments and posts, and pushing a database upward destroys everything written since your last pull.

Write that rule somewhere the whole team can see it. Almost every catastrophic WordPress deployment story is somebody pushing a database in the wrong direction.

Decide what actually goes in the repo

Three approaches, in increasing order of discipline and payoff. Pick one deliberately rather than drifting into the first.

1. Theme only. The repo contains one custom theme, sometimes a custom plugin. Everything else is installed through wp-admin. Easy to start, and it means you have no reproducible build: rebuilding the site from the repo is impossible.
2. All of wp-content, minus uploads. Themes, plugins and mu-plugins in git; core and media excluded. This is the pragmatic middle and where most teams land. You get reproducible plugin versions and a real diff when a plugin changes.
3. Composer-managed, Bedrock-style. Core, plugins and themes are all declared as dependencies. The repo holds a composer.json and your own code, nothing else. The lockfile is the source of truth and composer install rebuilds the site anywhere.

Option three is the one I reach for on anything with more than one developer, because “which plugin version is production running” becomes a question with an answer. The honest cost: premium plugins that are not on a Composer repository need either a private repository or a manual step, and that friction is real. If most of your plugin stack is commercial, option two is a reasonable place to stop.

One thing to check if you are following an older tutorial. The Composer repository landscape for WordPress plugins shifted recently: WPackagist was acquired by WP Engine, and the Roots team launched WP Packages as an independent alternative with different package naming. Both work. Any guide written before that change will use the older wpackagist-plugin/ prefix, so make sure the repository URL and the prefixes in your composer.json match each other.

The gitignore that matters

# Secrets and machine-specific config. Never committed.
wp-config.php
.env

# Content, not code. This gets pulled down from production,
# it never gets deployed up.
wp-content/uploads/

# Generated at runtime or by the build.
wp-content/cache/
wp-content/upgrade/
wp-content/debug.log
node_modules/
vendor/

Ignoring vendor/ assumes your deploy runs composer install. If your pipeline just rsyncs the repo with no build step, commit it instead, or you will ship a site with no dependencies. Decide which, then be consistent, because half-and-half is how you end up with a missing autoloader at two in the morning.

Close the second write path

This is the step people skip, and it is the one that makes everything else work. If production can still install and update plugins from the browser, your repo is not the source of truth. It is a suggestion.

// wp-config.php on production and staging.

// Removes the ability to install, update or delete plugins and
// themes from the admin. Also disables the built-in file editor,
// so DISALLOW_FILE_EDIT is implied by this one.
define( 'DISALLOW_FILE_MODS', true );

// Stops core from updating itself behind your back, including
// the automatic minor updates that would otherwise change files
// your deploy is about to overwrite.
define( 'AUTOMATIC_UPDATER_DISABLED', true );

Be clear-eyed about the trade-off, because it is a real one. You have just taken away automatic security updates. That is only an improvement if you replace them with something: a scheduled dependency update job, a weekly review of the update list, a bot that opens a pull request when a plugin version changes. If you disable updates and then do nothing, you have made the site less safe, not more controlled.

The update flow now runs through the repo. On your local machine:

# Composer-managed: bump the lockfile, commit it, deploy.
composer update wp-plugin/some-plugin

# wp-content-in-git: let WP-CLI do the download locally,
# then commit the resulting file changes as a normal diff.
wp plugin update some-plugin

The second form is worth appreciating. A plugin update becomes a reviewable diff. When something breaks two days later, git log tells you exactly which files changed and git revert puts them back.

Configuration per environment

Database credentials, salts and API keys differ per environment and none of them belong in git. Keep wp-config.php out of the repo and place it on each server, or use an environment file with a committed .env.example showing the required keys with no values.

Core has a first-class way to tell environments apart, and it is underused:

// Per environment, in wp-config.php.
// Recognised values: 'local', 'development', 'staging', 'production'.
define( 'WP_ENVIRONMENT_TYPE', 'staging' );

<?php
// mu-plugin, committed to the repo, safe on every environment
// because it checks where it is running.

if ( wp_get_environment_type() !== 'production' ) {

    // Keep non-production sites out of search results.
    add_filter( 'pre_option_blog_public', '__return_zero' );

    // Short-circuit wp_mail() entirely. Returning a non-null
    // value stops WordPress before it hands anything to PHPMailer.
    add_filter( 'pre_wp_mail', '__return_false' );
}

That mail filter is the single most valuable line in this post. Restore a production database onto staging, forget this, and the first cron run mails real order confirmations, password resets and abandoned-cart nudges to real customers from a URL that does not work. It is silent until it is very loud.

Moving the database in the allowed direction

Set up WP-CLI aliases once and the rest becomes short commands instead of a runbook nobody follows.

# wp-cli.yml in the project root, committed to the repo.
@staging:
  ssh: deploy@staging.example.com/var/www/staging.example.com
@production:
  ssh: deploy@example.com/var/www/example.com

Now pulling production down to your machine is one pipe. The trailing hyphen means “write the dump to standard output”, and the leading hyphen on the import means “read it from standard input”, so nothing touches disk:

# Back up your local database first, then overwrite it.
wp db export local-backup.sql
wp @production db export - | wp db import -

The URLs inside that dump still point at production, and they are buried in serialized PHP arrays where a plain SQL find-and-replace corrupts the string length prefixes. This is exactly what wp search-replace exists for: it unserializes, replaces, and reserializes properly.

# Always dry-run first and actually read the table list it prints.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid --dry-run

# Then for real.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid

wp cache flush

Two flags worth understanding rather than copying:

• --skip-columns=guid because the guid column is a permanent identifier that feed readers and external systems use to decide whether they have already seen a post. Rewriting it makes subscribers re-receive your entire archive, and you find out from an angry email weeks later.
• --all-tables-with-prefix because plugins create their own tables using your prefix, and those tables are not registered with WordPress. Without this, the replacement misses them and you get half-migrated URLs in places nobody checks.

Uploads are the other half of the content problem. Do not put them in git; a media library will outgrow a repository quickly and every clone pays for it. Sync them separately with rsync when you need them, or skip them entirely locally and let a plugin proxy missing images from production. Offloading media to object storage sidesteps the question completely and is worth considering on any site with a large library.

The deploy itself

Three broad options. git pull on the server is the simplest and the one I would move away from first: it puts a .git directory next to your document root, it has no build step, and a failed pull leaves the site half-updated. Rsync from CI is the pragmatic default. Atomic releases with a symlink swap, the way Deployer or Capistrano work, are the most correct because the site switches versions instantly and rollback is a symlink change.

A workable rsync deploy from GitHub Actions, with staging on merge and production on tag:

name: Deploy
on:
  push:
    branches: [main]     # goes to staging
    tags: ['v*']         # goes to production

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4    # pin to a full SHA in production

      - name: Build
        run: |
          composer install --no-dev --optimize-autoloader
          npm ci
          npm run build

      - name: Sync files
        run: |
          echo "$DEPLOY_KEY" > deploy_key
          chmod 600 deploy_key
          rsync -az --delete 
            -e "ssh -i deploy_key -o StrictHostKeyChecking=accept-new" 
            --exclude-from=.deployignore 
            ./ "$TARGET"
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
          TARGET: ${{ secrets.DEPLOY_TARGET }}

--delete is what makes the target match the source exactly, which is the entire point of a deploy. It is also the flag that will erase your uploads directory if .deployignore is wrong. Test the exclude list against staging before you ever point this at production, and use --dry-run the first time.

Files landing on the server is not the same as a working deploy. Run a post-deploy step over SSH:

#!/usr/bin/env bash
set -euo pipefail

cd /var/www/example.com

# Applies schema changes that a core or plugin version bump needs.
# Safe to run when there is nothing to do.
wp core update-db

# Rewrite rules live in the database, so a plugin that registers
# new routes needs this or you get 404s on the new URLs.
wp rewrite flush

# Options and transients are cached; stale ones survive a file deploy.
wp cache flush

# Prove the site answers before you call the deploy finished.
# --fail makes curl exit non-zero on a 5xx instead of shrugging.
curl --fail --silent --show-error https://example.com/ > /dev/null

If you run PHP-FPM with OPcache, add a reset to that script. Otherwise PHP happily serves the previous version of your code from its compiled cache and you spend twenty minutes convinced the deploy did not run.

Making staging safe to hold real data

Staging is only useful if it mirrors production, and the moment it does it becomes dangerous. Every one of these has bitten somebody:

• Outbound email. Covered above. Kill it at the wp_mail() level rather than trusting a plugin setting somebody can toggle.
• Cron. A restored production database brings production’s scheduled jobs with it. Set DISABLE_WP_CRON on staging and do not add a system cron job unless you specifically need to test scheduling.
• Payment gateways. The database carries live API keys. Overwrite them with test-mode credentials as part of your restore script, not by hand afterwards.
• Search indexing. Filter blog_public as shown, and put HTTP basic auth in front of the whole environment. Duplicate content ranking above your real site is an avoidable embarrassment.
• Personal data. A production restore is real customer data on a server that probably has weaker access controls. Either anonymise user records during the restore or treat staging with the same seriousness as production.

Write the restore as a single script that does all of this in order, so nobody has to remember step four. If it is a checklist in a wiki, it will be skipped.

Troubleshooting

The deploy ran but the site is unchanged

In order: OPcache serving compiled bytecode, a page cache or CDN holding the old response, or rsync excluding the directory you edited. Check them in that order. Confirm the file on disk actually changed before blaming anything upstream of it.

White screen immediately after deploying

Usually a missing autoloader, meaning vendor/ was neither committed nor built. Check the PHP error log rather than guessing. A memory limit lower on the server than locally is the other common cause, and it produces the same blank page.

Site works, admin redirects in a loop

siteurl and home in the options table disagree with the URL you are actually using. Almost always a search-replace that missed. Check with wp option get siteurl and wp option get home, and set them explicitly with wp option update rather than running the replacement again.

Serialized data broken after a migration

Somebody ran sed or a SQL REPLACE on the dump. The length prefix inside each serialized string no longer matches the string, so PHP silently fails to unserialize and widgets, theme options and ACF fields come back empty. Restore from the dump and redo it with wp search-replace. There is no partial recovery worth attempting.

Permalinks return 404 after deploy

Rewrite rules are stored in the database and did not get regenerated. Run wp rewrite flush. If it recurs on every deploy, add it to the post-deploy script permanently.

Plugin reverts to an older version after every deploy

Somebody is still updating in wp-admin. That is not a pipeline bug, it is the second write path being open. Set DISALLOW_FILE_MODS and give the team a documented way to request an update.

Common mistakes

• Committing wp-config.php, and with it your database password and salts.
• Putting wp-content/uploads in the repo, then wondering why clones take ten minutes.
• Pushing a staging database over production because “the content is the same”. It never is.
• Using sed or SQL to swap URLs instead of wp search-replace.
• Replacing the guid column, and re-sending your archive to every RSS subscriber.
• Running rsync --delete against production without testing the exclude list.
• Leaving updates enabled in wp-admin, so the repo and the server drift apart silently.
• Disabling automatic updates without putting a review process in its place.
• Restoring production data onto staging with mail still enabled.
• Forgetting DISABLE_WP_CRON on staging, so scheduled jobs fire twice across two environments.
• Treating “the files copied” as proof the deploy worked, with no smoke test.

Best practices

• Make the repo the only way code reaches a server, and enforce it with DISALLOW_FILE_MODS.
• Keep code moving forward and content moving backward, and state that rule where the team can see it.
• Deploy staging from a branch and production from a tag, so production releases are deliberate.
• Dry-run every search-replace and every rsync the first time.
• Back up the target database immediately before any deploy that touches it.
• Script the staging restore end to end, including mail, cron, keys and anonymisation.
• Finish every deploy with a real HTTP request through the public URL, not a file listing.
• Use environment-aware code with wp_get_environment_type() instead of commented-out blocks.
• Keep uploads out of git and sync or offload them separately.
• Replace automatic updates with a scheduled dependency review, and put it in a calendar.

FAQ

Should WordPress core be in the repository?

Only if you are not managing it another way. Committing core works and makes the repo self-contained, at the cost of noisy diffs on every update. Declaring core as a Composer dependency is cleaner and gives you a pinned version without the noise. What you should not do is leave core unmanaged, so nobody can say which version production runs.

How do I handle the database in a Git workflow?

You do not put it in Git. Git handles code; the database is handled by scripted pulls in one direction. Production to staging to local, with a search-replace on import. The only time a database legitimately goes upward is the initial launch, before there is real content to lose.

What about plugin settings changed in staging?

This is the genuinely hard part and there is no clean answer. Settings live in the database, so they do not deploy. Options are: reapply them by hand in production with a documented checklist, script them as WP-CLI commands in a deploy step, or use a plugin that exports configuration to files. Whichever you pick, write it down, because “I’ll remember” is how a setting gets applied to staging and never production.

Is Bedrock necessary?

No. It is a well-made set of defaults for Composer-managed WordPress with environment-based configuration, and it saves assembling those pieces yourself. Plenty of teams run a perfectly disciplined Git deployment with a standard directory layout. Adopt it if you want the conventions; do not adopt it as a prerequisite.

Do I need three environments?

You need local and production. Staging earns its cost when you have multiple people, a client who signs off on changes, or a site where breakage costs money. On a personal blog it is overhead. On a store, skipping it means production is your test environment.

Can I do this on shared hosting?

Partly. You need SSH and ideally WP-CLI, and plenty of shared plans have neither. Some managed WordPress hosts offer a git push deployment target, which handles the file half for you. If you want the full workflow with your own build steps and post-deploy hooks, a small VPS from a provider like InterServer gives you the SSH access and cron control the workflow assumes.

How do I roll back a bad deploy?

Files are easy: redeploy the previous tag, or swap the symlink back if you use atomic releases. The database is the hard part, which is why you take a dump immediately before any deploy that runs a migration. Rolling files back without rolling the database back can leave you worse off than the broken version, so know which of the two actually changed before you act.

The one thing to remember

Git-based WordPress deployment is not really a pipeline problem. The pipeline is the easy part, and you can build a working one in an afternoon. The hard part is making the repo the only way code reaches a server, and then keeping the database flowing in exactly one direction.

Get those two right and everything else is detail. Get them wrong and you will keep having the same confusing morning, staring at a plugin version that nobody remembers changing.

Want this set up properly on your site?

Retrofitting version control onto a live WordPress site is fiddly work, mostly because you have to do it without downtime and without losing whatever has been changed directly on the server. Things I take on:

• Moving an existing production site into Git without downtime, including reconciling whatever has drifted on the server.
• Converting a site to Composer-managed plugins and core, with a working lockfile and a sane update process.
• Building local, staging and production environments that actually match, with per-environment configuration.
• Deploy pipelines in GitHub Actions or GitLab CI: build, rsync or atomic releases, post-deploy WP-CLI steps, smoke test, rollback path.
• Scripted database refreshes from production to staging with mail, cron, API keys and personal data handled automatically.
• Server setup for the workflow: SSH deploy users, correct file ownership, OPcache resets, cron.

Tell me how your site is hosted and how plugins currently get updated, and I will tell you what the first step should be.

The post Two Sources of Truth: Git-Based WordPress Deployment From Local to Production appeared first on John Nessime.

That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer’s token, pushed a malicious commit, and then retroactively repointed every version tag at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner’s secrets into the build log. On public repositories, those logs are readable by anyone.

Nobody’s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.

A GitHub Actions VPS deployment is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.

Who can actually reach that key

Worth being explicit, because it is a longer list than people expect:

• Anyone with write access to the repository. A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.
• Every action in the job. Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.
• Anyone who can influence what runs. This is why pull_request_target combined with checking out the pull request’s own code is such a well-known foot-gun: it runs the fork’s code with the base repository’s secrets in scope.

The goal is not to make the key unreachable. It is to make a stolen key not worth much.

Decide how the runner authenticates

Three options, and the right one depends on what you already run.

1. A dedicated SSH key in repository secrets. The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.
2. An overlay network. Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.
3. A self-hosted runner on the server. No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.

One thing that sounds appealing and is not practical: allowlisting GitHub’s runner IP ranges at your firewall. The ranges are published through GitHub’s meta API, but they are enormous and they change, and every other GitHub customer’s job runs from inside them. It buys you very little for a lot of maintenance.

Narrow the key until it is boring

Assume the key leaks. What can the holder do? Work backwards from there.

Give it its own user. A deploy account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.

Restrict the key in authorized_keys. SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:

# ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions

The guard script decides what the key is allowed to do. Keep it short enough to read in one go:

#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" >&2
    exit 1
    ;;
esac

Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.

Scope the sudo entry too. The deploy user needs to reload a service, not run arbitrary commands as root:

# visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted deploy user with no sudo is a reasonable stopping point.

Verify the server, not just the key

Almost every tutorial on this topic contains StrictHostKeyChecking=no, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.

# Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com

Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.

The workflow

name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" > ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" > ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz > /dev/null

Three details in there worth calling out, because they are the ones people leave out.

-o BatchMode=yes stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.

--delete is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if .deployignore is wrong. Run it once with --dry-run against staging and read the list before you point it at production.

The final curl --fail is the difference between “the files copied” and “the site works”. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.

Third-party actions

There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain ssh and rsync for that reason.

Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.

Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.

Gate production properly

The environment: key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:

• Scoped secrets. The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.
• Required reviewers. The job pauses and waits for a named human to approve it. This is the single control that turns “anyone with write access can deploy” into “anyone with write access can request a deploy”.
• Deployment branch and tag rules. Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.

Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.

For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.

Troubleshooting

Permission denied (publickey)

The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses printf '%sn' rather than echo: it guarantees the newline regardless of what got stored.

After that, check permissions on the server: ~/.ssh must be 700 and authorized_keys 600, owned by the deploy user, and the home directory must not be group-writable. Add -vvv to the ssh command to see how far the handshake gets, and read journalctl -u ssh on the server for the reason, since the client is deliberately vague.

Host key verification failed

Either the known_hosts variable is empty, or the server’s host key genuinely changed. If you rebuilt the server, regenerate it with ssh-keyscan and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.

The job hangs until the timeout

SSH is waiting on a prompt nobody will answer. Add -o BatchMode=yes and it fails immediately with a readable reason instead.

rsync works from my laptop, fails from the runner

If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records $SSH_ORIGINAL_COMMAND, run the job once, and read what rsync actually sent.

Works on push, fails on pull requests

Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for pull_request_target to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.

A secret appeared in the logs

Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.

Common mistakes

• Using an existing personal SSH key instead of generating a dedicated one for the runner.
• Giving the deploy user full sudo, or a sudo rule with a wildcard in it.
• Disabling host key checking with StrictHostKeyChecking=no.
• Referencing third-party actions by tag in the job that holds your production key.
• Interpolating ${{ secrets.X }} directly into a run: script instead of passing it through env:.
• Leaving default write permissions on a job that only needs to read.
• Using cancel-in-progress: true on a deploy, so a second push interrupts the first mid-copy.
• Running rsync --delete at production without testing the exclude list.
• Treating “the files copied” as a successful deploy, with no health check.
• Deploying straight from branch pushes with no approval step.
• Relying on log masking to keep a leaked secret out of the output.
• Never rotating the deploy key, including after someone leaves the team.

Best practices

• Dedicated deploy user, dedicated key, used for nothing else.
• Restrict the key in authorized_keys with restrict and a forced command.
• Scope sudo to the exact commands the deploy needs.
• Pin every action to a full commit SHA and let Dependabot bump them.
• Keep the deploy job small: fewer steps means fewer things that can read the secret.
• Set permissions: explicitly, defaulting to read-only.
• Verify host keys from a stored known_hosts value.
• Use environments with required reviewers and deployment rules for production.
• Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.
• End every deploy with a real HTTP request against the public URL.
• Rotate the deploy key on a schedule and whenever someone leaves.

FAQ

Is it safe to store an SSH private key in GitHub secrets?

Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.

Should I use a ready-made SSH deploy action?

They work and they are convenient. My preference is plain ssh and rsync in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.

Can I use OIDC instead of a stored key?

Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub’s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.

Should I allowlist GitHub’s IP ranges on my firewall?

Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer’s runners share those ranges. Narrowing what the key can do achieves more.

Self-hosted runner or hosted runner?

Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.

How do I roll back a bad deploy?

Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.

How often should the deploy key be rotated?

On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to authorized_keys and updating the secret, so there is no good reason for it to be a project.

The one thing to remember

The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.

So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger’s code did with your credentials.

Want this built or reviewed?

Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:

• Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.
• Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.
• Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.
• Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.
• Pinning and auditing third-party actions, plus repository settings that limit what can run.
• Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.

Send me your workflow file and the output of ssh deploy@yourhost 'id', and I will tell you what a stolen key would get.

The post Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully appeared first on John Nessime.

You go back through the logs and there it is, four hundred lines up, in a step that reported success: a download that returned an HTML error page, got piped into tar, and produced an archive containing nothing useful. The shell reported the exit status of tar, not of curl. The build carried on. The pipeline was never lying on purpose. It just never asked the right question.

Two things go wrong with pipelines, and only one of them gets talked about. The famous one is slow feedback: thirty-four minutes of build and integration tests before a linter tells you a file is missing a trailing newline. The expensive one is a pipeline that passes when it should have stopped. A fail fast pipeline fixes the first problem. Making it explain itself fixes the second, and that is the half most teams skip.

This post covers both: how to order stages so real failures surface in seconds, how to stop the shell from hiding non-zero exits, how to design failure output that hands the next engineer the fix, and the troubleshooting steps for when a pipeline stubbornly refuses to fail at the right moment.

The failure that actually bites: a pipeline that lies

Start here, because everything else is optimisation on top. A slow pipeline wastes your afternoon. A silently passing pipeline ships a broken artifact and burns a day of debugging in production, usually while somebody is on a call with a customer.

The shell is where most of this happens. Bash gives you three defaults that are wrong for CI, and one line fixes all three.

#!/usr/bin/env bash
set -euo pipefail

# Without pipefail this line exits 0 even when curl fails,
# because the shell reports the status of the LAST command
# in the pipe, which is tar.
curl -sSL https://example.com/artifact.tar.gz | tar -xz

What each flag actually protects

• -e stops the script when a command exits non-zero. Without it, a failed database migration is followed by a successful echo, and the script exits 0 because the last command succeeded.
• -u turns an unset variable into an error. This is the one that saves you from rm -rf "$PREFIX/build" expanding to rm -rf "/build" when someone renames an environment variable in the pipeline settings and forgets the script.
• -o pipefail makes a pipeline return the first non-zero status instead of the last one. Every curl | tar, cat | jq, and docker logs | grep in your scripts is currently reporting only the exit code of the thing on the right.

The exit status is only half the story though. Plenty of tools return 0 while doing something useless, and curl is the classic offender.

# curl exits 0 here even when the server returns 500,
# and you end up untarring an HTML error page.
curl -sSL "$ARTIFACT_URL" | tar -xz

# --fail          makes curl exit non-zero on 4xx and 5xx
# --show-error    keeps the reason visible even with --silent
# --location      follows redirects instead of saving a 302 body
curl --fail --silent --show-error --location "$ARTIFACT_URL" | tar -xz

Now the download failing stops the build, and pipefail makes sure that failure propagates past the pipe. The first fixes the check, the second makes sure the check is heard.

Where set -e quietly gives up

This trips people up constantly. set -e is suspended whenever a command’s exit status is being consumed by something else. It is not a safety net, it is a default.

set -e

# -e does NOT stop the script here.
# The if statement consumes the exit status.
if grep -q "TODO" src/main.go; then
  echo "found"
fi

# -e does NOT stop the script here either.
# The left side of || is explicitly allowed to fail.
run_migrations || echo "migrations failed, continuing anyway"

# -e DOES stop the script here.
run_migrations

The same trap applies inside containers. On Debian-based images /bin/sh is dash, which has no pipefail option at all, so putting set -o pipefail inside a RUN line fails outright. Set the shell at the top of the Dockerfile instead. Hadolint flags this as DL4006 and it is worth wiring into your static stage.

# Set this before any RUN instruction that contains a pipe.
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN curl --fail --silent --show-error --location 
      https://example.com/tool.tar.gz | tar -xz -C /usr/local/bin

Order stages by cost, not by category

Most pipelines are laid out the way the team thinks about work: tests, then build, then security, then deploy. That grouping is comfortable and it is the wrong axis. Order by two things instead, cost and failure probability.

The rule I use: every stage should be cheaper than the one after it and more likely to fail than the one after it. If a check catches ten percent of your broken commits and takes fifteen seconds, it goes in front of the check that catches two percent and takes nine minutes. That is the whole design principle behind a fail fast pipeline.

1. Syntax and static analysis. shellcheck, hadolint, yamllint, terraform validate, tflint, formatter checks, type checks. Seconds, no dependencies, no network. This stage catches an embarrassing share of broken pushes.
2. Unit tests. No database, no containers, no network. If they need any of those, they are integration tests and belong further down.
3. Secrets and dependency scanning. Fast, and a leaked credential is worth stopping before you build an image around it.
4. Build. This is where you start spending real minutes and cache space.
5. Integration and end-to-end. Slow, flaky, expensive to run. Everything above should have already caught the boring failures.
6. Deploy and post-deploy verification.

A note on the trade-off, because this ordering costs something. Splitting the fast checks into their own stage adds a checkout and a container pull before your unit tests even start. On a busy shared runner that overhead is real. If your static stage takes twelve seconds of work and forty seconds of setup, merge it into the test job and run it as the first step instead. The principle is failure order, not job count.

The other thing that keeps the fast lane fast is caching, and it is the piece people get wrong in the opposite direction: they cache so aggressively that a stale dependency tree hides a genuine break. Cache the package manager’s download directory, not the resolved dependency tree, and key the cache on the lockfile hash.

Let the platform kill work you already know is doomed

Ordering stages is only useful if the platform actually stops when something breaks. Three settings do most of the work, and they exist under different names on every runner.

GitHub Actions

concurrency:
  # A new push to the same branch cancels the run that is
  # already in flight. Nobody needs results for a commit
  # that has been superseded.
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  static:
    runs-on: ubuntu-latest
    timeout-minutes: 5          # a hung job is worse than a failed one
    steps:
      - uses: actions/checkout@v4   # pin to a full commit SHA in production
      - run: shellcheck scripts/*.sh
      - run: hadolint Dockerfile

  test:
    needs: static               # will not start until static passes
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false          # see note below
      matrix:
        suite: [unit, contract]
    steps:
      - uses: actions/checkout@v4
      - run: make test-${{ matrix.suite }}

That fail-fast: false looks like it contradicts the whole post, so it is worth explaining. Matrix fail-fast defaults to true: one leg fails, the rest get cancelled. That is right when the legs are redundant and you only need to know that something broke. It is wrong when the legs tell you different things. If your matrix is three runtime versions and only one fails, cancelling the others hides whether this is a version-specific problem or a general one, and you will burn a rerun finding out. Fail fast on the pipeline, not on the diagnosis.

timeout-minutes is the setting people skip and regret. A job waiting on an interactive prompt, a dead package mirror, or a container that never becomes healthy will sit there consuming a runner until someone notices. Set a timeout on every job, generous enough that a slow day does not trip it and tight enough that a hang gets caught inside one coffee break.

GitLab CI

stages: [static, test, build, deploy]

default:
  interruptible: true     # allows auto-cancel when a newer pipeline starts
  retry:
    max: 2                # capped at 2 by GitLab, so 3 attempts total
    when:
      # Retry only on infrastructure problems.
      # Never retry script_failure: that is your code failing,
      # and retrying it just hides a flaky test.
      - runner_system_failure
      - stuck_or_timeout_failure
      - api_failure

lint:
  stage: static
  timeout: 5m
  script:
    - shellcheck scripts/*.sh

integration:
  stage: test
  needs: [lint]           # DAG: starts as soon as lint passes
  script:
    - make integration

The retry: when list is the important bit and it is the one people get lazy about. A bare retry: 2 retries on everything including a genuine test failure, which converts an intermittent bug into an invisible one. Scope retries to the failure classes that are genuinely the infrastructure’s fault. Jenkins has failFast true for parallel stages, Buildkite and CircleCI have their own equivalents, and the reasoning transfers cleanly.

A failure should hand you the answer, not the search

Here is the test I apply to a pipeline failure. Can someone who did not write the change, at an inconvenient hour, fix it from the failure output alone without opening the source? If not, the failure message is incomplete.

Four things need to be in there:

• What failed, named precisely. Not “build failed” but the command and its exit code.
• Why, in terms of the input rather than the internals. “lockfile is out of date with package.json”, not “npm ci exited 1”.
• Where, meaning the file, the line, or the config key that caused it.
• What to do next. One concrete action, even if it is just a runbook link.

In shell, a small helper plus an ERR trap gets you most of the way there.

#!/usr/bin/env bash
set -euo pipefail

fail() {
  echo "ERROR: $1" >&2
  echo "  fix: $2" >&2
  exit 1
}

# Catches anything you did not write an explicit check for,
# and tells you the line and the command that blew up.
trap 'echo "FAILED at line $LINENO: $BASH_COMMAND" >&2' ERR

[[ -f config/app.yaml ]] || fail 
  "config/app.yaml is missing" 
  "copy config/app.yaml.example and set DATABASE_URL"

[[ -n "${IMAGE_TAG:-}" ]] || fail 
  "IMAGE_TAG is empty" 
  "check the tag job output; it sets IMAGE_TAG from the git SHA"

Note ${IMAGE_TAG:-} rather than $IMAGE_TAG. Under set -u the bare form aborts with a shell error before your nice message ever prints, which is technically correct and practically useless.

On GitHub Actions you can go one step further and attach the error to the file and line, so it shows up as an annotation on the pull request instead of something to be dug out of a log.

# Renders as a red annotation on that exact line in the diff view.
echo "::error file=deploy/values.yaml,line=42::image tag is empty"

# Visible, but does not fail the job.
echo "::warning file=Dockerfile,line=7::apt cache not cleaned"

Two more habits worth building. Upload logs, test reports, and screenshots as artifacts on failure, conditionally, so the evidence survives the job that produced it. And emit test results as JUnit XML so the platform renders a list of failed test names instead of asking people to scroll. Both are cheap and both cut the time between “it broke” and “I know why” more than any clever retry logic will.

Exit codes are a contract, so use more than one

Most scripts have two states: 0 and “something went wrong”. That collapses distinctions the pipeline needs. Terraform gets this right and it is a good model to copy.

# -detailed-exitcode changes the meaning of the exit code:
#   0 = succeeded, no changes
#   1 = error
#   2 = succeeded, changes present
set +e
terraform plan -detailed-exitcode -out=tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift. Skipping apply." ;;
  2) echo "Changes detected. Holding for approval." ;;
  *) echo "Plan failed. See log above." >&2; exit "$code" ;;
esac

The set +e around it is deliberate: with -e active, exit code 2 would kill the script before you got to interpret it. This is one of the few places where turning the flag off is the correct move, and it is worth a comment in the file explaining why so nobody “fixes” it later.

Do the same in your own scripts. Reserve a code for “the thing you asked for is not possible” versus “I could not reach the API” versus “your input is wrong”. The pipeline can then retry one and refuse to retry the others, which is the difference between a retry that absorbs a runner hiccup and a retry that hides a race condition for six months.

Gate on what you care about, not what is easy to check

This is the deploy-stage version of the same lie. Plenty of commands confirm that a request was accepted, not that the thing you wanted happened.

# Returns 0 the moment the API server accepts the manifest,
# even if every pod goes into CrashLoopBackOff a second later.
kubectl apply -f k8s/

# This is the step that actually waits for the rollout,
# and fails the job if it does not converge.
kubectl rollout status deployment/api --timeout=180s

# And this proves the application answers a request,
# not just that a pod reports Ready.
curl --fail --silent --show-error https://api.example.com/healthz

Run that last check against the public hostname, through your CDN, rather than against the pod directly. If you front the app with Cloudflare or similar, a smoke test that skips the edge will happily pass while real users hit a cached error page or a misrouted origin. Same reasoning applies to the origin host itself: if the app runs on a VPS from a provider like InterServer behind a reverse proxy, verify through the proxy, because that is the path traffic takes.

Then track how long each stage takes over time. Pipeline duration and failure rate per stage are perfectly ordinary metrics to ship into Prometheus and graph in Grafana, or into whatever platform you already use for application monitoring. When the fast lane creeps from forty seconds to four minutes, that is a slow regression nobody will notice from a single run.

Troubleshooting

The job passed but the artifact is wrong

Look for pipes first. Grep the scripts for | and check whether pipefail is set in that shell. Then check for tools that return 0 on failure: curl without --fail, docker run where the entrypoint swallows errors, anything wrapped in || true that somebody added during an incident and never removed.

set -e is set but the script keeps going

The failing command is almost certainly inside a condition, on the left of || or &&, or in a function that is itself being called from an if. Add set -x temporarily and read the trace to see exactly where control flow continued. Then use an explicit check with your fail helper rather than relying on -e at that point.

Works locally, fails in CI (or the reverse)

Usually a shell difference. Your terminal runs bash or zsh, the runner may execute the script with sh. Add an explicit shebang, and on GitHub Actions set the shell on the step or at the job level rather than assuming. Inside containers, remember dash has no pipefail and no [[ ]].

The job hangs until the platform kills it

Something is waiting on stdin. Package managers prompting for confirmation, ssh asking about a host key, a database client waiting for a password. Add the non-interactive flags, set CI=true where tools respect it, and put a timeout on the job so the next occurrence fails in ten minutes instead of sixty.

Fails on the first run, passes on rerun

Resist the retry. Log the failure with enough context to identify it later, quarantine the test into a non-blocking job if it is genuinely blocking the team, and give it an owner and a date. A blanket retry turns a flaky test into a permanent tax on everyone’s confidence in the pipeline.

The failure output is a wall of noise

Suppress progress bars and spinners in CI, they render as thousands of unreadable lines. Then print a short summary block at the very end of the job, after the noise, listing what failed and what to do. People read the bottom of a log first.

Common mistakes

• Writing set -e without -o pipefail and assuming the script is covered.
• Using || true to silence a step that fails intermittently, instead of finding out why.
• Running the slowest, flakiest checks before the ones that take fifteen seconds.
• Retrying on every failure class rather than only infrastructure faults.
• No job timeouts, so a hang consumes a runner until someone notices.
• Treating kubectl apply or a deploy API call as proof the deploy worked.
• Failure messages that name the internal function that threw, and nothing the reader can act on.
• Caching so aggressively that a stale dependency tree hides a real break.
• Discarding logs and test reports when the job fails, which is exactly when you need them.
• Turning off matrix fail-fast everywhere by reflex, so every broken commit costs the full matrix.

Best practices

• Put set -euo pipefail at the top of every shell script the pipeline runs, and set the Dockerfile SHELL before any piped RUN.
• Order stages so each one is cheaper and more likely to fail than the next.
• Give every job a timeout, and cancel superseded runs on the same branch.
• Use explicit checks with actionable messages instead of relying on -e to catch everything.
• Distinguish exit codes so the pipeline can react differently to different failures.
• Verify deploys with a rollout status check plus a real HTTP request through the public path.
• Upload artifacts and structured test reports on failure.
• Pin action and image references to immutable digests so a green pipeline stays reproducible.
• Track stage duration and failure rate over time, the same way you track application metrics.
• Fix the failure message the first time somebody has to ask what it means.

FAQ

What does “fail fast” actually mean in a CI/CD pipeline?

Two things. First, the pipeline stops as soon as it knows the result will be a failure, rather than finishing work whose outcome no longer matters. Second, the cheapest checks that are most likely to catch a problem run first, so the average failure surfaces in seconds. The second half is a design choice about ordering, not a platform setting.

Is set -e enough on its own?

No. It is suspended inside conditions, on the left side of && and ||, and after !. It also does nothing about pipes, which is why -o pipefail matters, or about commands that return 0 while failing, which is why explicit checks matter. Use it as a baseline, not as your only line of defence.

Should I disable matrix fail-fast?

Only when the legs give you different diagnostic information. If the matrix is runtime versions or operating systems and you need to know whether the break is universal or specific, let all legs run. If the legs are interchangeable shards of the same test suite, leave fail-fast on and save the compute.

How do I stop a pipeline from passing when a step really failed?

Work through three things in order. Set pipefail so pipes propagate failure. Add the flags that make tools return non-zero on failure, such as curl --fail. Then replace implicit trust with explicit assertions: after a deploy, check that the rollout converged and that the service answers a request. Most silent passes come from one of those three gaps.

Are automatic retries a good idea?

Only for failures that are the infrastructure’s fault: runner crashes, image pull errors, API timeouts from the CI platform itself. Never retry a script failure by default. And make retries visible, because a job that quietly succeeds on the third attempt is telling you something important that nobody sees.

How long should a fast feedback stage take?

Short enough that an engineer waits for it rather than switching context. Somewhere under a couple of minutes for the combined lint, format, and unit stage is a reasonable target for most codebases. Past that, people push and walk away, and you lose the benefit of fast feedback even though the pipeline is technically quick.

Does this apply to self-hosted runners too?

All of it, and timeouts matter more. On hosted runners a stuck job eventually hits the platform’s own ceiling. On your own machine it holds a slot until you intervene, so a hung job blocks the queue for everyone. Set explicit timeouts and make sure the runner cleans up its workspace between jobs.

The one thing to take away

Speed is the part everyone optimises and honesty is the part that actually costs money. A fail fast pipeline that finishes in ninety seconds is worth very little if one of those seconds contains a swallowed exit code. Fix the lying first: pipefail, tools that return non-zero when they fail, and assertions that check the outcome rather than the request. Then reorder for speed.

And judge every failure by one question. Could someone else fix this from the output alone, without reading the source? If the answer is no, the pipeline has not finished its job, no matter how quickly it went red.

Need help fixing a pipeline that lies to you?

If any of this sounds like your CI, this is the kind of work I take on. Typical engagements look like:

• Auditing an existing GitHub Actions, GitLab CI, or Jenkins pipeline for silently swallowed failures and false-green stages.
• Restructuring stage order and caching so the common failure surfaces in seconds instead of after the build.
• Hardening deploy verification: rollout gates, smoke tests through the real public path, and automatic rollback on a failed health check.
• Rewriting pipeline shell scripts with proper error handling, exit code contracts, and failure messages an on-call engineer can act on.
• Wiring pipeline duration and failure metrics into Prometheus and Grafana so regressions get caught early.

Send me the workflow file and a link to a failing run, and I will tell you what I would change first.

The post Green Check, Broken Build: How to Build a Fail Fast Pipeline That Explains Why appeared first on John Nessime.

None of that is a bug. AWS optimises the empty account for getting something running in ten minutes. The problem is that most accounts never get corrected afterwards, and eighteen months later there is production traffic sitting on top of decisions nobody made deliberately.

An AWS account baseline is the set of things you lay down before the first workload, so the account starts from a defensible posture instead of a blank console. This walks through the decisions that actually matter, why each one is the way it is, and what it costs. I have put a working reference implementation on GitHub — terraform-aws-baseline — so you can read the real Terraform rather than take my word for any of it.

What a baseline is, and what it is not

A baseline is not a landing zone in the Control Tower sense, and it is not a compliance framework. It does not make you SOC 2 compliant and nothing that fits in one repository will.

What it is: the boring foundational layer that everything else sits on. State storage, network segmentation, identity, and the guardrails that stop a careless afternoon becoming an incident. If you get these right once, every workload that lands afterwards inherits them for free.

The structure that works is three layers. A bootstrap stack that creates the state backend and runs exactly once. Reusable modules for the things every environment needs. Thin per-environment roots that compose those modules with different parameters — dev gets one NAT gateway and seven-day log retention, prod gets one per availability zone and a year.

Decision one: no long-lived AWS credentials, anywhere

Static access keys in CI are the single most common way AWS accounts get compromised. They live in a secrets store, they get copied into a second one, somebody echoes one into a build log, and nobody rotates them because rotation breaks things.

GitHub Actions can assume an AWS role using a short-lived OIDC token that GitHub signs and AWS verifies. There is no key to store, nothing to leak, and nothing to rotate.

The important part is the trust policy, and one word in it decides whether this is a security improvement or a vulnerability:

data "aws_iam_policy_document" "ci_assume" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # StringEquals, not StringLike. This is the whole ballgame.
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values = [
        "repo:${var.github_org}/${var.github_repo}:ref:refs/heads/${var.github_branch}"
      ]
    }
  }
}

Swap StringEquals for StringLike and add an asterisk, and you have widened that trust from one branch of one repository to whatever the wildcard covers. Get sloppy enough — a bare repo:* — and any repository on GitHub can assume your role. This is not theoretical; it is a recurring finding in real audits.

Scope it to one repository and one branch. If you need more, add more statements, explicitly.

Decision two: treat Terraform state as a secret, because it is

Terraform state contains every attribute of every resource it manages. Database passwords, generated keys, connection strings. Marking a variable sensitive redacts it from console output and changes nothing about what is written to the state file.

So the state bucket gets four things: server-side encryption with a customer-managed KMS key that rotates, versioning so a corrupted state can be recovered, all four public access block flags, and a bucket policy that refuses plaintext connections outright.

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "deny_insecure_transport" {
  statement {
    sid       = "DenyInsecureTransport"
    effect    = "Deny"
    actions   = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]

    principals {
      type        = "*"
      identifiers = ["*"]
    }

    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }
}

All four public access block flags, not two. They cover different paths — ACLs and bucket policies, existing objects and future ones — and setting a subset leaves a gap that reads as protected on a dashboard.

There is a chicken-and-egg problem here that catches everyone: the stack that creates the state bucket cannot store its own state in a bucket that does not exist yet. You run it once with local state, then migrate:

cd bootstrap
terraform init
terraform apply
terraform output          # bucket, lock table, KMS key, region

# Now move bootstrap's own state into the bucket it just created
terraform init -migrate-state

Decision three: shut the default security group

Every VPC gets a default security group that permits all traffic between members and all outbound traffic. Anything launched without an explicit security group lands on it. That is a quiet failure mode — nothing errors, the instance works, and it has more network reach than anyone intended.

Terraform can adopt and empty it:

resource "aws_default_security_group" "default" {
  vpc_id = aws_vpc.this.id

  # No ingress and no egress blocks. Terraform strips AWS's rules,
  # so anything that accidentally lands here can reach nothing.
}

Four lines, and misconfiguration becomes loud instead of silent.

Decision four: three subnet tiers, and the bottom one has no way out

The standard two-tier split — public and private — leaves databases in the same tier as application servers. Both can reach the internet through NAT. That is fine right up until something in the data tier is compromised and quietly exfiltrates over 443, which looks like ordinary egress traffic.

• Public — NAT gateways and load balancers. Route to the internet gateway.
• Private — workloads. Outbound through NAT, no inbound from the internet.
• Database — data stores. No default route at all.

The third tier is enforced by absence rather than by a rule:

resource "aws_route_table" "database" {
  vpc_id = aws_vpc.this.id

  # Deliberately no 0.0.0.0/0 route. Not to the IGW, not to NAT.
  # Nothing here can reach the internet, and nothing can reach it.
}

No security group to misconfigure, no NACL to get wrong. The route simply does not exist. That is a much stronger guarantee than a rule someone can widen during an incident and forget to narrow afterwards.

Decision five: gateway endpoints, the one genuinely free win

S3 and DynamoDB gateway endpoints cost nothing. Not “cheap” — zero. They add a route so traffic to those services stays on the AWS backbone rather than going out through NAT.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.this.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  route_table_ids = concat(
    aws_route_table.private[*].id,
    aws_route_table.database[*].id,
  )
}

Two benefits, and the second is the one people miss. Traffic to S3 never traverses NAT, so you stop paying per-gigabyte processing on it. And the isolated database tier can still reach S3 and DynamoDB despite having no internet route, because the endpoint is a route within the VPC rather than a path out of it.

Your Terraform state lives in S3. Without this endpoint, every plan and apply pushes state through the NAT gateway and you are billed for the privilege.

Decision six: SSM Session Manager, not SSH

Port 22 open anywhere means key pairs to distribute, a bastion host to patch, and an audit trail that is whatever auth.log happens to contain.

resource "aws_iam_role_policy_attachment" "ssm_core" {
  role       = aws_iam_role.instance.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

That one attachment gets you shell access with no inbound ports, no key pairs, no bastion. Access is IAM-authorised, so revoking someone is a permissions change rather than a key rotation, and every session can be logged centrally.

Worth knowing the caveat: SSM agents reach the SSM endpoints over the internet by default, so instances in the private tier still need NAT for this. Removing NAT entirely means adding interface endpoints for ssm, ssmmessages and ec2messages, and those cost roughly seven dollars a month per availability zone each.

Decision seven: pin actions to commit SHAs

A Git tag is a movable pointer. If an action’s repository is compromised, an attacker can repoint v4 at whatever they like, and every workflow referencing that tag runs it on the next build — with your OIDC role attached.

steps:
  # A tag can be moved. A commit SHA cannot.
  # Dependabot updates these pins and shows you the diff.
  - uses: actions/checkout@<full-40-character-sha>   # v4.2.2

The comment matters as much as the pin. Without it nobody can tell at a glance which version they are looking at, and the pin becomes something people update blindly.

What it actually costs

Almost all of it is NAT gateways. Everything else is rounding.

• NAT gateways — roughly $32 a month each, plus about $0.045 per gigabyte processed. One in dev, one per availability zone in prod.
• VPC, subnets, route tables, internet gateway — free.
• S3 and DynamoDB gateway endpoints — free.
• KMS keys — about a dollar a month each.
• Flow logs — around $0.50 per gigabyte ingested. Retention is the lever: seven days in dev, a year in prod.
• State bucket and lock table — cents.

Ballpark, that lands around $35 a month for dev and $105 for prod, dominated entirely by NAT. If you only need the isolated tiers and nothing requires outbound internet, dropping NAT takes dev close to free.

Per-AZ NAT in production is an availability decision, not a security one. A single NAT gateway is a single point of failure for outbound traffic across every AZ that routes through it.

Gates that run before any credentials exist

Most of what goes wrong in Terraform is catchable statically. Formatting, syntax, provider misuse, and known-insecure patterns can all be found without touching an AWS account, which means they can run on every commit and in CI without the pipeline holding any cloud credentials at all.

# On every commit, locally
pre-commit install        # gitleaks, fmt, validate, tflint, checkov, terraform-docs

# The same gates CI runs
make all                  # fmt + validate + tflint + checkov

Running the identical command locally and in CI is the part that matters. When those two drift, people stop trusting the local check and start pushing to see what the pipeline says, and the fast feedback loop you built quietly stops existing.

Note also what is missing: there is no terraform plan in that pipeline. Plan needs real credentials, and a static-only pipeline is one that cannot leak anything. A production setup would add a plan job running under the OIDC role and posting to the pull request — but that is a deliberate step up in blast radius, not a default.

What a baseline like this deliberately leaves out

Being clear about scope is more useful than pretending completeness. The reference implementation covers preventative controls and stops there. Missing, on purpose:

• Detective controls. No CloudTrail, Config, GuardDuty or Security Hub. A complete account baseline includes them; this is the preventative half.
• Compute. There is an instance role and profile, but nothing uses them. No ASG, no ECS or EKS, no load balancer.
• Interface endpoints. Only the free gateway ones. Fully private SSM needs paid interface endpoints.
• Multi-region and DR. Versioning covers state recovery. There is no cross-region replication and no failover story.
• Policy-as-code tests. Checkov catches known misconfigurations; there are no Terratest or Conftest assertions on module contracts.

A baseline you can describe the edges of is more useful than one advertised as complete. You know exactly what you still have to build.

Common mistakes

• Using StringLike with a wildcard in an OIDC trust policy. Scope to one repository and one branch.
• Assuming sensitive = true keeps values out of state. It does not.
• Setting two of the four S3 public access block flags and considering the bucket locked down.
• Leaving the default security group as AWS shipped it.
• Putting databases in the same tier as application servers because two tiers felt like enough.
• Skipping gateway endpoints and paying NAT processing charges on your own state file.
• Referencing actions by tag and assuming a tag is immutable.
• Committing backend.hcl or terraform.tfvars. State bucket names embed the account ID.

Frequently asked questions

Is an AWS account baseline the same as a landing zone?

No. A landing zone in the Control Tower sense governs many accounts — organisational units, service control policies, account vending. A baseline is what you lay down inside a single account. Baselines are what you need when you have one or two accounts and Control Tower would be enormous overkill.

Why not just use Control Tower?

Control Tower is a good answer for a multi-account organisation with governance requirements. It is a lot of machinery for a team running a handful of accounts, and it hides the decisions rather than making them legible. Writing the baseline yourself means you can read every choice and change any of them.

Do I really need a separate database subnet tier?

It costs nothing and closes an entire exfiltration path. If your data tier has no route to the internet, a compromised database cannot phone home regardless of what the attacker does with it. Two tiers is not wrong, but the third is close to free.

How do I cut the NAT gateway cost?

Use one NAT for non-production instead of one per AZ. Add gateway endpoints so S3 and DynamoDB traffic bypasses NAT entirely. Beyond that, work out what genuinely needs outbound internet — often less than you assume — and consider dropping NAT for environments that do not.

Can I use this with OpenTofu?

Yes. It is standard HCL against the AWS provider, so it runs on either. Swap terraform for tofu in the commands and everything else is the same.

Where does the OIDC provider itself live?

It is account-global, so only one environment can create it and the others reference it. That is workable but awkward. In a real organisation it belongs in a separate account-level stack that runs before any environment.

Wrapping up

None of the decisions in an AWS account baseline are individually clever. Scope the trust policy. Encrypt the state. Empty the default security group. Give the data tier no way out. Take the free endpoints. Use SSM instead of SSH. Pin your actions. Each one is a few lines of Terraform.

What makes them valuable is doing them before the first workload arrives, because retrofitting network segmentation into a running account is a project, and doing it at the start is an afternoon.

The full implementation is on GitHub at JohnNessime/terraform-aws-baseline, MIT licensed. It is a reference implementation rather than something that has served production traffic — a worked example of how I would lay an account down, with the reasoning written out in the docs. Read it, disagree with parts of it, take what is useful.

Need a baseline laid down properly?

I work with teams on AWS foundations — account structure, network segmentation, IAM that does not rely on long-lived keys, Terraform layouts that survive more than one engineer, and CI pipelines that catch problems before they reach an account.

If you would rather have this done once and done right, you can hire me on Upwork.

The post A Secure-by-Default AWS Account Baseline in Terraform appeared first on John Nessime.