Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between “the server works” and “the server is reproducible” turns into a weekend.

This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client’s credentials from ever touching another client’s box, and how to prove a build is finished instead of assuming it.

The four failure families worth designing against

Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.

1. You cannot rebuild it

The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run apt install, every file edited over SSH at 1am, every “I’ll document that later” is a piece of the machine that exists nowhere but the machine.

This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.

2. Every client is a slightly different server

Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.

Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts “all our clients have…” becomes a guess.

3. One credential, too much blast radius

This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client’s server, and the same API token provisions every client’s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.

Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.

4. The handover you cannot perform

Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.

How to automate server provisioning for new clients

The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.

Layer 1: the machine itself

Terraform or OpenTofu talks to your provider’s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.

Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client’s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.

Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.

Layer 2: first boot with cloud-init

cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.

#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo

Two deliberate omissions. There is no package_upgrade: true, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in runcmd is a script you will never test again.

python3 is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.

Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:

cloud-init schema --config-file user-data.yaml

Layer 3: convergence with Ansible

This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.

The layout that scales across clients separates the shared build from the per-client differences:

infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml

Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.

Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:

cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  > ../inventory/hosts.ini

There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.

Layer 4: the verification gate

This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.

Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:

- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost

Note ssh_unit as a variable rather than a literal: the SSH unit is ssh.service on Debian and Ubuntu but sshd.service on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.

The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.

Keeping clients apart: secrets and access

Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client’s infrastructure is a single, contained action.

For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.

# .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...

Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.

For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.

If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.

Troubleshooting a provisioning run

Ansible connects before cloud-init has finished

The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.

Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:

cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log

One caveat that catches people: never put cloud-init status --wait inside a runcmd or bootcmd block. It will deadlock waiting for the process that is running it.

cloud-init appears to have done nothing

Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in /var/lib/cloud/instance/sem/ are preventing a rerun.

# what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot

Never run cloud-init clean on a production host. It removes cached state and the following boot will behave as a first boot.

Host key verification failed on a rebuilt server

Rebuild a machine on the same IP and every client with the old key in known_hosts refuses to connect. The tempting fix is to disable host key checking in ansible.cfg. Do not. That turns a real integrity check into a permanent no-op across every client you manage.

Remove the stale entry and pin the new one as part of the provisioning flow:

ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 >> ~/.ssh/known_hosts

Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.

The playbook reports changes on every single run

Almost always a shell or command task with no creates, changed_when or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.

Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:

ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff

Common mistakes

• Putting the application build in runcmd. It runs once, never gets tested, and is invisible to every later change.
• One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.
• Copying a role into a client directory to make one small change, instead of extracting a variable.
• Treating “the playbook exited zero” as proof the server is ready. It is proof no task threw.
• Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.
• Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.
• Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.

Best practices

• Rebuild before you need to. Destroy and rebuild a client’s staging host on a schedule. It is the only honest test of reproducibility.
• Version your roles and pin them per client. A shared role that everyone tracks on main means one commit changes six clients at once.
• Keep first boot minimal. If it can wait for Ansible, it should.
• Make monitoring part of provisioning. Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.
• Same for backups. An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.
• Separate credentials per client, always. Keys, tokens, encryption keys, state backends.
• Write the runbook as you build. The provisioning repo’s README is the handover document. If it is accurate, the engagement can end cleanly.

Frequently asked questions

Do I need Terraform if I only manage a few VPS instances?

Not necessarily. If you are creating three servers a year through a provider’s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.

Should I build custom images instead of provisioning at boot?

Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.

Terraform or OpenTofu for client projects?

Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.

How do I automate server provisioning without exposing SSH to the internet?

The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.

How do I stop configuration drift once the server is live?

Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.

What belongs in cloud-init versus Ansible?

cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.

How long should provisioning a new client server take?

Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.

The one thing worth remembering

When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer “can you rebuild this?” with yes, on any day, without opening a shell on the existing machine.

Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.

Start with one client. Get their staging box rebuilding from scratch. Then make the second client’s build reuse the roles from the first. That is the whole path.

Need a provisioning setup you can actually hand over?

Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:

• Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime
• Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets
• Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works
• Writing the verification playbook that decides whether a build is finished, and wiring it into CI
• Fixing idempotency so check-mode runs become a usable drift report instead of noise
• Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves

If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.

The post Automating Server Provisioning for New Clients Without Building a Snowflake Farm appeared first on John Nessime.

The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.

This post covers how to keep Kubernetes secrets out of Git in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.

The failure that actually bites: encrypted in Git, wide open in the cluster

Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run kubectl get secret -o yaml in that namespace reads the value in one pipe through base64 -d.

So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster’s own blast radius.

Three things need to be true alongside whichever tool you pick:

• Encryption at rest is on. A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an EncryptionConfiguration, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.
• RBAC on secrets is actually restrictive. A depressing number of clusters grant blanket get and list on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.
• Secrets are mounted as files, not shoved into environment variables where you can help it. Environment variables leak through /proc/<pid>/environ, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.

Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:

# Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default

If that returns yes, your Git hygiene is not the weakest link in the chain.

Already leaked? Rotate first, rewrite second, and understand what rewriting does not do

Order matters here more than technique. GitHub’s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker’s ability to use it. History surgery is the optional follow-up, not the fix.

The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues’ clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.

When you do rewrite, use git filter-repo. It has replaced filter-branch as the recommended tool and it is dramatically faster on any repo with real history.

# Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags

Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A git pull will happily drag the old objects back in.

Two families of solution, and how to tell which one you are in

Every approach to this problem is either encrypt the value and commit the ciphertext, or commit a pointer and fetch the value at runtime. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.

Sealed Secrets: the lowest-friction way to start

A controller in the cluster holds an RSA private key and publishes the matching certificate. The kubeseal CLI encrypts a Secret manifest against that certificate and produces a SealedSecret custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.

# Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  > pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='<value>' 
  --dry-run=client -o yaml > db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml < db-auth.yaml > db-auth-sealed.yaml

rm db-auth.yaml

Because you can hand out pub-cert.pem, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.

Two details that catch people. First, sealing is scoped by default: a SealedSecret is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with --scope namespace-wide or --scope cluster-wide if you genuinely need to. Second, the Helm chart names the controller sealed-secrets while the CLI looks for sealed-secrets-controller by default, so --controller-name is not optional in practice.

Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.

SOPS with age: encrypt files, not just Kubernetes objects

SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with age, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.

A .sops.yaml at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:

creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient

The encrypted_regex line is the one worth understanding rather than copying. Without it SOPS encrypts everything including metadata, and you lose the readable-diff property that made this approach attractive. Restricting it to data and stringData keeps names, namespaces and labels reviewable in a pull request.

Flux decrypts SOPS natively: you point a Kustomization at a Secret holding the age private key via decryption.provider: sops and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.

SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain .env. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.

External Secrets Operator: stop putting the value in Git at all

The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password

This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for refreshInterval, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.

The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.

One live upgrade trap. The external-secrets.io/v1beta1 API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to v1, which produced permanent drift in Argo CD for anyone whose Git manifests still said v1beta1. Argo sees v1 in the cluster, Git says v1beta1, and it reconciles forever. Update your manifests to v1 before you cross that boundary, not after.

Secrets Store CSI Driver: when a Secret object is one object too many

Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.

The key you forgot to back up

This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.

Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every SealedSecret in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.

• Back up the sealing key out of band. The controller’s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.
• Same for age. The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.
• Add a second recipient for anything that matters. SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.
• Know your re-key procedure before you need it. For SOPS that is sops updatekeys across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.

Stopping plaintext from reaching the repo in the first place

All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.

1. Put a scanner in the pre-commit hook. gitleaks and trufflehog both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.
2. Run the same scan in CI, on the whole history, not just the diff. The diff-only scan misses everything that predates the day you added the scanner.
3. Turn on server-side push protection. GitHub’s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone’s local setup.
4. Make the plaintext path short-lived by construction. Never write a plaintext manifest to a tracked path. Use /tmp, or pipe straight into the encrypt step so the value never lands on disk at all.
5. Add *.dec.yaml, secret.yaml and friends to .gitignore as a convention. Not a control, just one less way to be careless.
6. Remember your shell history. A --from-literal with a real password ends up in ~/.bash_history in cleartext. Use --from-file, or read from a variable, or prefix the command with a space if your shell is configured to skip those.

Troubleshooting the four things that go wrong

The SealedSecret applies cleanly but no Secret appears

Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.

# The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml

The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.

The ExternalSecret sits there doing nothing

The status conditions carry the real message, and the events under describe usually name the provider error directly.

kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend

Check the store before the secret. If the SecretStore is not Ready, nothing referencing it will ever sync, and the error you want is on the store, not the ExternalSecret. When using a ClusterSecretStore, remember that any secret reference inside it needs an explicit namespace, because a cluster-scoped object has no namespace of its own to fall back on.

Argo CD or Flux reports permanent drift

Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO v1beta1 removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.

SOPS decryption fails only in the cluster

You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in .sops.yaml but running sops --encrypt with an explicit --age flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file’s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.

Common mistakes

• Rewriting history before rotating the credential, and treating the rewrite as the remediation.
• Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.
• Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.
• Never backing up the sealing key or the age identity.
• Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.
• Encrypting the entire manifest with SOPS instead of just data and stringData, and losing every useful diff.
• Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.
• Piping a Helm release’s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.

How I would actually decide

Skip the feature matrix and answer three questions in order.

1. Do you already run a secret store? If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.
2. Do you have config outside Kubernetes? Terraform variables, Ansible vars, application .env files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.
3. Neither? Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.

Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.

FAQ

Is base64 in a Kubernetes Secret a form of encryption?

No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.

Should I use Sealed Secrets or External Secrets Operator?

If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.

Do I still need to rewrite Git history after rotating a leaked credential?

Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.

Can Argo CD decrypt SOPS files on its own?

Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.

Is it safe to commit the Sealed Secrets certificate or the age public key?

Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.

The one thing worth remembering

If you take one idea away, make it this: the work to keep Kubernetes secrets out of Git and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of SealedSecret resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.

Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for filter-repo, because rotation is the part that actually stops the bleeding.

Need help sorting out secrets in your cluster?

This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:

• Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output
• Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure
• Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover
• Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions
• Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces
• Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives

If you have a controller log, a stuck ExternalSecret, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.

The post Keep Kubernetes Secrets Out of Git Without Kidding Yourself appeared first on John Nessime.