That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource’s name inside the module. The author had tidied up a resource "aws_db_instance" "this" into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.

That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.

This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.

The bill for a module arrives at upgrade time

Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: module.database.aws_db_instance.this. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.

For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.

The fix has existed for a long time and it is genuinely good: the moved block. A module author who renames something declares the rename in code, ships it with the release, and every consumer’s next plan updates state instead of destroying infrastructure.

# Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}

The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its moved blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.

So here is the whole thesis, and everything below is elaboration: a module is worth reusing if the next version bump is cheap. Everything else is secondary.

Five signals that reusable Terraform modules are worth adopting

These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.

1. It keeps state addresses stable, or tells you when it does not

Open the repository. Search for moved. Read the changelog for the last major release and see whether renames are called out.

A module with no moved blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says “v3.0 renames the launch template resource, here are the moved blocks” is telling you it has thought about consumers.

2. The input surface is narrow and typed

Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.

Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.

variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}

Three things are doing work there. optional() with a default means a caller who wants the simple case writes two lines instead of six. nullable = false means a caller cannot pass null and get a confusing failure three resources deep. The validation block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.

That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.

3. You can pin it, and the pin means something

This trips people up because the two common source types behave differently.

# Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}

The ~> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as ~> 5.0.1 it accepts patches only. Which one you want depends entirely on whether you trust the author’s reading of semantic versioning, and the honest answer for third-party modules is usually “not enough to auto-accept minors in production”.

For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at init time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.

4. It has tests you can run without a cloud account

Terraform’s built-in test framework uses .tftest.hcl files and the same HCL you already write. The critical detail for module authors is that a run block set to command = plan creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.

# tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}

Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.

OpenTofu runs the same format through tofu test, and additionally recognises a .tofutest.hcl extension for tests that should only run there. Useful if you maintain modules that target both tools.

5. Somebody answers issues

Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.

Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when init refuses to resolve a version constraint.

Three kinds of module, and which ones actually pay off

Not everything that can be a module should be. It helps to name the three shapes you see in the wild.

The thin wrapper

One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.

There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.

The composition module

Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.

Most modules worth reusing live here.

The landing zone

An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.

Public modules or your own?

The case for public modules is stronger than the “we have specific needs” crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.

The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.

My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.

There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp’s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.

For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.

Making your own modules worth reusing

If you are on the authoring side, this is the order I would do things in.

1. Write it inline first, twice. Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module’s interface, discovered rather than guessed.
2. Extract with moved blocks in the same commit. Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.
3. Type the inputs properly. Objects with optional() defaults over flat strings, nullable = false where null makes no sense, validation blocks with error messages that name the fix.
4. Prefer for_each over count for anything collection-shaped. With count, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.
5. Generate the docs. terraform-docs reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.
6. Lint and scan on every PR. terraform fmt -check, terraform validate, tflint for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.
7. Tag releases, and mean it. A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.

One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.

Troubleshooting the plans that scare you

“Forces replacement” after a version bump

Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.

terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'

If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version’s source. A rename with no moved block is the usual answer, and you can write the moved block yourself in your root configuration as a workaround.

A module with a provider block inside it

Modules should declare which providers they need, not configure them. A module that contains a configured provider block rather than just a required_providers entry limits what callers can do with it, including using for_each or count on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.

Changes to the module source are not picked up

Remote modules are downloaded into the working directory during init. Changing a ref or a version constraint in your configuration does not re-download anything on its own. Run terraform init -upgrade. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.

Registry version list looks stale

A registry’s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry’s view after any change to that integration.

Common mistakes

• Pinning to a branch. ?ref=main is not a pin. It is an unannounced deployment scheduled for whenever someone next runs init.
• Modularising after one use. You end up encoding the first use case’s assumptions as the abstraction, then bolting flags onto it for every case after.
• Copying environments instead of parameterising them. Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.
• Passing every provider argument straight through. If a module’s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.
• Deleting moved blocks too early. They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.
• Applying a plan you did not read. Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.

How I decide whether to adopt a module

Roughly ten minutes, in this order:

1. Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.
2. Grep for moved. Its presence tells me the author thinks about consumers.
3. Read the last major release notes. Are breaking changes explained, with an upgrade path?
4. Check for a tests directory and whether CI runs on pull requests.
5. Look at the issue tracker’s recent activity, not its total count.
6. Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.

And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.

Frequently asked questions

How many input variables is too many for a Terraform module?

There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.

Should I use public modules or write my own?

Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else’s.

Do Terraform modules work with OpenTofu?

Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.

How do I pin a module version from a Git source?

The version argument only applies to registry sources. For Git, append a ref query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at init time, which defeats the purpose of pinning entirely.

What happens if a module removes a resource in a new version?

Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.

Can I test a Terraform module without cloud credentials?

Yes, for a useful subset. Run blocks set to command = plan evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.

Do I need a private registry to share modules internally?

No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.

Conclusion

Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.

The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.

Need a second pair of eyes on your module library?

I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:

• Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations
• Adding moved blocks and upgrade notes to in-house modules so version bumps stop being incidents
• Narrowing wide module interfaces with typed objects, optional() defaults and validation rules that fail with useful messages
• Setting up module CI: fmt, validate, tflint, policy scanning and generated docs on every pull request
• Writing terraform test suites that run on plan, including negative tests that prove your validation rules actually fire
• Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all

Send me a module’s variables.tf and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.

The post Terraform Modules Worth Reusing (And the Ones That Cost You) appeared first on John Nessime.

Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.

This post covers how to keep Kubernetes secrets out of Git in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.

The failure that actually bites: encrypted in Git, wide open in the cluster

Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run kubectl get secret -o yaml in that namespace reads the value in one pipe through base64 -d.

So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster’s own blast radius.

Three things need to be true alongside whichever tool you pick:

• Encryption at rest is on. A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an EncryptionConfiguration, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.
• RBAC on secrets is actually restrictive. A depressing number of clusters grant blanket get and list on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.
• Secrets are mounted as files, not shoved into environment variables where you can help it. Environment variables leak through /proc/<pid>/environ, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.

Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:

# Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default

If that returns yes, your Git hygiene is not the weakest link in the chain.

Already leaked? Rotate first, rewrite second, and understand what rewriting does not do

Order matters here more than technique. GitHub’s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker’s ability to use it. History surgery is the optional follow-up, not the fix.

The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues’ clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.

When you do rewrite, use git filter-repo. It has replaced filter-branch as the recommended tool and it is dramatically faster on any repo with real history.

# Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags

Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A git pull will happily drag the old objects back in.

Two families of solution, and how to tell which one you are in

Every approach to this problem is either encrypt the value and commit the ciphertext, or commit a pointer and fetch the value at runtime. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.

Sealed Secrets: the lowest-friction way to start

A controller in the cluster holds an RSA private key and publishes the matching certificate. The kubeseal CLI encrypts a Secret manifest against that certificate and produces a SealedSecret custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.

# Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  > pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='<value>' 
  --dry-run=client -o yaml > db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml < db-auth.yaml > db-auth-sealed.yaml

rm db-auth.yaml

Because you can hand out pub-cert.pem, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.

Two details that catch people. First, sealing is scoped by default: a SealedSecret is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with --scope namespace-wide or --scope cluster-wide if you genuinely need to. Second, the Helm chart names the controller sealed-secrets while the CLI looks for sealed-secrets-controller by default, so --controller-name is not optional in practice.

Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.

SOPS with age: encrypt files, not just Kubernetes objects

SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with age, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.

A .sops.yaml at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:

creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient

The encrypted_regex line is the one worth understanding rather than copying. Without it SOPS encrypts everything including metadata, and you lose the readable-diff property that made this approach attractive. Restricting it to data and stringData keeps names, namespaces and labels reviewable in a pull request.

Flux decrypts SOPS natively: you point a Kustomization at a Secret holding the age private key via decryption.provider: sops and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.

SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain .env. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.

External Secrets Operator: stop putting the value in Git at all

The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password

This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for refreshInterval, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.

The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.

One live upgrade trap. The external-secrets.io/v1beta1 API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to v1, which produced permanent drift in Argo CD for anyone whose Git manifests still said v1beta1. Argo sees v1 in the cluster, Git says v1beta1, and it reconciles forever. Update your manifests to v1 before you cross that boundary, not after.

Secrets Store CSI Driver: when a Secret object is one object too many

Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.

The key you forgot to back up

This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.

Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every SealedSecret in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.

• Back up the sealing key out of band. The controller’s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.
• Same for age. The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.
• Add a second recipient for anything that matters. SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.
• Know your re-key procedure before you need it. For SOPS that is sops updatekeys across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.

Stopping plaintext from reaching the repo in the first place

All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.

1. Put a scanner in the pre-commit hook. gitleaks and trufflehog both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.
2. Run the same scan in CI, on the whole history, not just the diff. The diff-only scan misses everything that predates the day you added the scanner.
3. Turn on server-side push protection. GitHub’s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone’s local setup.
4. Make the plaintext path short-lived by construction. Never write a plaintext manifest to a tracked path. Use /tmp, or pipe straight into the encrypt step so the value never lands on disk at all.
5. Add *.dec.yaml, secret.yaml and friends to .gitignore as a convention. Not a control, just one less way to be careless.
6. Remember your shell history. A --from-literal with a real password ends up in ~/.bash_history in cleartext. Use --from-file, or read from a variable, or prefix the command with a space if your shell is configured to skip those.

Troubleshooting the four things that go wrong

The SealedSecret applies cleanly but no Secret appears

Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.

# The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml

The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.

The ExternalSecret sits there doing nothing

The status conditions carry the real message, and the events under describe usually name the provider error directly.

kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend

Check the store before the secret. If the SecretStore is not Ready, nothing referencing it will ever sync, and the error you want is on the store, not the ExternalSecret. When using a ClusterSecretStore, remember that any secret reference inside it needs an explicit namespace, because a cluster-scoped object has no namespace of its own to fall back on.

Argo CD or Flux reports permanent drift

Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO v1beta1 removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.

SOPS decryption fails only in the cluster

You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in .sops.yaml but running sops --encrypt with an explicit --age flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file’s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.

Common mistakes

• Rewriting history before rotating the credential, and treating the rewrite as the remediation.
• Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.
• Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.
• Never backing up the sealing key or the age identity.
• Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.
• Encrypting the entire manifest with SOPS instead of just data and stringData, and losing every useful diff.
• Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.
• Piping a Helm release’s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.

How I would actually decide

Skip the feature matrix and answer three questions in order.

1. Do you already run a secret store? If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.
2. Do you have config outside Kubernetes? Terraform variables, Ansible vars, application .env files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.
3. Neither? Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.

Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.

FAQ

Is base64 in a Kubernetes Secret a form of encryption?

No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.

Should I use Sealed Secrets or External Secrets Operator?

If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.

Do I still need to rewrite Git history after rotating a leaked credential?

Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.

Can Argo CD decrypt SOPS files on its own?

Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.

Is it safe to commit the Sealed Secrets certificate or the age public key?

Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.

The one thing worth remembering

If you take one idea away, make it this: the work to keep Kubernetes secrets out of Git and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of SealedSecret resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.

Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for filter-repo, because rotation is the part that actually stops the bleeding.

Need help sorting out secrets in your cluster?

This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:

• Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output
• Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure
• Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover
• Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions
• Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces
• Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives

If you have a controller log, a stuck ExternalSecret, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.

The post Keep Kubernetes Secrets Out of Git Without Kidding Yourself appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.

That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.

Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.

This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.

What Docker Compose genuinely gets right

Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.

• The whole system fits in one file you can read. Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.
• Dev and prod can share a base file. Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.
• No control plane to operate. A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.
• Recovery is comprehensible. When something breaks at 2am, docker compose ps and docker compose logs tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.

If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.

Failure family one: the deploy gap

This is the one that surprises people who came from a platform that did rolling updates for them.

docker compose up -d is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.

The deploy.update_config block with order: start-first exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.

What to do instead

Three options, in increasing order of effort:

1. Accept the gap and shrink it. Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.
2. Put a reverse proxy in front and run two slots. Traefik, Caddy or plain Nginx in a container, with app-blue and app-green services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.
3. Drain at the edge. If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.

Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:

# Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.

That last line matters more than it looks. A service with no health check is invisible to --wait, so a deploy script can report success while your API is still crash-looping.

Failure family two: the disk fills up and nothing tells you

Docker’s default logging driver is json-file, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local driver is the recommended alternative because it rotates out of the box and uses a more compact format.

So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.

Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}

Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.

You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:

x-logging: &default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    <<: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    <<: *default-logging

Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.

# Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v

Failure family three: state you did not mean to keep

Two related traps here.

The first is anonymous volumes. If an image declares a VOLUME in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.

The second is docker compose down. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.

The practical rule: on a production host, restarting services is docker compose up -d or docker compose restart. down belongs in a maintenance procedure with a fresh backup, not in a daily habit.

Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.

Failure family four: ports that bypass your firewall

This is the one that turns into a security incident rather than an outage.

When you write ports: - "5432:5432", Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status output that looks perfectly reassuring.

The fix is to stop publishing what does not need publishing:

services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]

Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.

While you are in daemon config: set default-address-pools as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.

Failure family five: secrets in environment variables

Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.

The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under /run/secrets/:

services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.

One more thing that trips people up: .env and env_file are different mechanisms. The .env file in the project directory feeds variable interpolation inside the Compose file. env_file passes variables into the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.

Failure family six: health checks that report but never act

A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.

So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.

Health checks are still worth defining, because they gate startup ordering and they gate --wait. Just be clear about what they do not do.

services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s

Note restart: unless-stopped rather than always. The difference shows up after a host reboot: always will start a container you had deliberately stopped, unless-stopped respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.

To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.

Troubleshooting a Compose stack that is misbehaving

The commands I reach for, roughly in order:

# The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' <container>

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"

Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.

Common mistakes

• Deploying :latest. You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.
• Keeping the version: key at the top of the file. Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.
• No resource limits anywhere. One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.
• Running docker compose up from an SSH session and walking away. Without -d the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.
• The Compose file lives only on the server. If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.
• Pulling from Docker Hub anonymously in a deploy script. Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.
• Automatic image updates on production. Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.

Best practices for Docker Compose in production

• One base file plus a production override. Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then docker compose -f compose.yaml -f compose.prod.yaml up -d is your deploy, and the diff between environments is a file you can read.
• Give every service a health check. It gates startup order, it makes --wait meaningful, and it gives you a status worth alerting on.
• Set memory and CPU limits on everything. Watch docker stats under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.
• Segment your networks. An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.
• Back up volumes, and restore one. A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.
• Make the deploy a script, not a memory. Pull, up with --wait, verify, prune. Six lines in the repo beats six commands somebody half-remembers.
• Run docker compose config in CI. It catches a malformed override before it reaches the server, and it costs nothing.

When to stop using Compose

Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.

The signals I treat as a real trigger, rather than as an excuse to rewrite everything:

• You need more than one machine for availability, not just for capacity.
• You need automatic rescheduling when a host dies, without somebody logging in.
• You are scaling components independently and often enough that doing it by hand is a real cost.
• Several teams deploy to the same infrastructure and need isolation from each other.

If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.

Frequently asked questions

Is Docker Compose production ready?

For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.

How do I get zero-downtime deployments with Docker Compose?

Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.

Does docker compose down delete my database?

Not by itself. Plain down removes containers and the project’s networks and leaves named volumes in place. Adding -v removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.

Why does my container keep restarting with exit code 137?

137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats for growth over time and dmesg for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.

Should I still write version: at the top of my Compose file?

No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.

Can I use Docker secrets without Swarm?

Yes, using file-based secrets. Declare a top-level secrets block with a file: source, grant it to a service, and the content appears at /run/secrets/<name> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect.

How many services is too many for one Compose file?

There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.

The one thing worth remembering

Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.

Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.

Need a second pair of eyes on your Compose setup?

I work with teams running containerised workloads on their own servers. Things I am usually brought in for:

• Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first
• Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI
• Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does
• Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall
• Volume backup and restore that has been tested by restoring, not just by running
• An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down

If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.

The post Docker Compose in Production: What Works and What Quietly Burns You appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was Delete, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.

That is the sharp edge of Kubernetes persistent volumes, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.

This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.

Three objects, one disk

The mental model that prevents most of this:

• StorageClass is the recipe. It says which driver provisions volumes and what the defaults are.
• PersistentVolume is the actual piece of storage in the cluster. Usually created for you rather than by you.
• PersistentVolumeClaim is the request a pod makes, and the thing that binds one-to-one with a PV.

The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:

# Read the RECLAIMPOLICY column. On managed clusters it usually
# says Delete, and nothing anywhere warns you about that.
kubectl get storageclass

For anything holding real data, define your own class with Retain and use it deliberately:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain              # PV outlives the PVC
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3

The trade-off is real and worth stating: Retain means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.

Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:

kubectl patch pv "$PV_NAME" 
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

ReadWriteOnce does not mean one pod

It means one node. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.

For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:

spec:
  accessModes:
    - ReadWriteOncePod        # enforced by the scheduler

The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.

spec:
  strategy:
    type: Recreate            # old pod goes first, then the new one

That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.

Volumes pin pods to a zone

Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in Pending after a node replacement or a scale-up that happened to land elsewhere.

kubectl describe pod tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.

The preventive setting is volumeBindingMode: WaitForFirstConsumer, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.

StatefulSet or Deployment

The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same volume, which is only correct when you have one replica.

So: one instance with a disk can be a Deployment with Recreate. Anything that scales and keeps per-replica state should be a StatefulSet.

StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to Retain:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain       # deleting the StatefulSet keeps the data
    whenScaled: Delete        # scaling down cleans up the extra replicas

That pairing is the sensible default for most workloads. Be careful with whenScaled: Delete on anything where you scale down temporarily and expect the data back when you scale up again.

Two one-way doors

Volumes grow, they do not shrink. Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.

A volume snapshot is not a database backup. Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.

Troubleshooting

Pod stuck in Pending. Read the events at the bottom of kubectl describe pod. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.

PVC stuck in Pending. kubectl describe pvc names the provisioner error. With WaitForFirstConsumer this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.

Rollout hangs forever on a stateful app. ReadWriteOnce plus rolling update. Switch to Recreate.

PVC will not delete. A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.

Volume is full but the PVC says it has room. You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.

Common mistakes

• Running production data on the default StorageClass without checking its reclaim policy.
• Assuming ReadWriteOnce prevents two pods from writing.
• A ReadWriteOnce volume on a Deployment with rolling updates.
• Scaling a Deployment with a PVC past one replica.
• Leaving volumeBindingMode: Immediate, then debugging Pending pods for an hour.
• Treating volume snapshots as database backups.
• Over-sizing a PVC on the assumption you can shrink it later.
• Removing a stuck finalizer instead of finding what is holding the volume.
• Deleting a namespace without checking what claims live in it.

Best practices

• Your own StorageClass with Retain for anything you would miss, and the default for everything else.
• WaitForFirstConsumer on every class you define.
• allowVolumeExpansion: true from the start, because adding it later does not help existing volumes.
• ReadWriteOncePod where exactly one writer is a correctness requirement.
• StatefulSets for per-replica state, single-replica Deployments with Recreate for everything else.
• An explicit persistentVolumeClaimRetentionPolicy rather than relying on the default.
• Application-level backups for databases, tested by restoring one.
• Alerts on volume capacity, because a full disk is the most predictable outage there is.

FAQ

Does deleting a PVC delete my data?

If the reclaim policy is Delete, yes, and that is the default nearly everywhere. Run kubectl get storageclass and read the RECLAIMPOLICY column before you assume otherwise.

Can two pods share a ReadWriteOnce volume?

Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.

Why is my pod Pending with a bound PVC?

Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in kubectl describe pod say so. WaitForFirstConsumer prevents it happening again.

Should I run databases in Kubernetes at all?

You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.

Can I shrink a PersistentVolumeClaim?

No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.

The one thing to remember

Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.

Go and run kubectl get storageclass on your production cluster now. If the reclaim policy on the class holding your database says Delete, you have found something worth fixing before lunch.

Need a hand with stateful workloads?

Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:

• Auditing a cluster’s storage classes, reclaim policies and access modes, and telling you what a stray kubectl delete would actually destroy.
• Moving stateful applications onto StatefulSets with sane retention, expansion and scheduling behaviour.
• Backup and restore design for data in Kubernetes, including an actual restore test rather than a documented intention.
• Diagnosing Pending pods, stuck rollouts and volumes that will not detach.
• CSI driver setup and storage class design on EKS and self-managed clusters.
• Capacity alerting so a full volume is a notification instead of an incident.

Send me the output of kubectl get storageclass and kubectl get pv, and I will tell you what stands out.

The post It’s Only a Claim: Kubernetes Persistent Volumes and Stateful Workloads appeared first on John Nessime.