<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Deployment | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/deployment/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/deployment/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 18 Oct 2026 06:03:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Deployment | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/deployment/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Dockerfile Layer Caching: Why Every Build Reinstalls Everything</title>
		<link>https://john-nessime.com/blog/technical-guides/dockerfile-layer-caching/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 18 Oct 2026 06:03:01 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Build Cache]]></category>
		<category><![CDATA[BuildKit]]></category>
		<category><![CDATA[Cache Mounts]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Dockerfile]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[Layer Caching]]></category>
		<category><![CDATA[Multi-Stage Builds]]></category>
		<category><![CDATA[Performance Tuning]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=409</guid>

					<description><![CDATA[<p>Your Dockerfile copies the lockfile before the source, and CI still reinstalls every dependency on every commit. That gap between a correctly ordered Dockerfile and a build that behaves as if it isn't comes down to five distinct failure families: ordering traps, a build context that poisons checksums, a cache that never reaches CI, cache mounts that vanish with the runner, and multi-stage export modes. Here is how invalidation actually works and how to prove which one is costing you.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dockerfile-layer-caching/">Dockerfile Layer Caching: Why Every Build Reinstalls Everything</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline log scrolls past and there it is again. The dependency install step, running in full, on a commit that touched one template file. Nobody edited the lockfile. The Dockerfile copies the lockfile before the source, exactly the way every guide says to. And it still reinstalls everything, every single time.</p>



<p class="wp-block-paragraph">That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn&#8217;t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it&#8217;s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.</p>



<p class="wp-block-paragraph">So this is organized by failure family rather than by feature. Ordering that looks right and isn&#8217;t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the <code>COPY --link</code> argument. Then how to prove which one you&#8217;re hitting instead of guessing.</p>



<h2 class="wp-block-heading">How the builder actually decides to reuse a layer</h2>



<p class="wp-block-paragraph">Two different rules are running, and mixing them up causes half the confusion.</p>



<p class="wp-block-paragraph">For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. <code>RUN apt-get update</code> matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.</p>



<p class="wp-block-paragraph">For <code>COPY</code>, <code>ADD</code>, and <code>RUN</code> instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh <code>git clone</code> on a CI runner doesn&#8217;t invalidate every copy by itself.</p>



<p class="wp-block-paragraph">Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.</p>



<p class="wp-block-paragraph">Three consequences catch people out. A <code>RUN</code> line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and &#8220;just rebuild the image&#8221; is not a security response on its own. Build arguments do invalidate the cache, so an <code>ARG</code> carrying a commit SHA near the top of the file is expensive. And build secrets do <em>not</em>: their contents are excluded from the checksum, though their IDs and mount paths are part of it.</p>



<p class="wp-block-paragraph">One more that bites teams chasing reproducible builds. <code>WORKDIR</code> respects the <code>SOURCE_DATE_EPOCH</code> build argument. Set it to a git commit timestamp for provenance and you invalidate <code>WORKDIR</code> and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.</p>



<h2 class="wp-block-heading">Failure family one: ordering that looks right</h2>



<p class="wp-block-paragraph">The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build</code></pre>



<p class="wp-block-paragraph">What breaks this in real repositories is usually not the copy order. It&#8217;s something above it that nobody thinks of as a layer.</p>



<ul class="wp-block-list">
<li><strong>An <code>ARG</code> near the top.</strong> A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.</li>



<li><strong>Labels carrying commit metadata.</strong> <code>LABEL</code> lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.</li>



<li><strong>Wildcard copies that pick up more than you think.</strong> <code>COPY package*.json ./</code> looks tidy until a generated file starts matching the glob. Name the files explicitly.</li>



<li><strong>Monorepo manifests scattered across packages.</strong> If installing needs twelve manifest files from twelve directories, copy them individually or accept you&#8217;re copying a subtree and lean hard on <code>.dockerignore</code>.</li>
</ul>



<p class="wp-block-paragraph">The general test: walk the file top to bottom and ask, for each line, &#8220;what makes this change?&#8221; Anything answering &#8220;every commit&#8221; belongs below anything expensive.</p>



<h2 class="wp-block-heading">Failure family two: the build context poisons the checksum</h2>



<p class="wp-block-paragraph">When you run <code>COPY . .</code>, the checksum covers everything in the context that isn&#8217;t excluded. That means <code>.git</code>, <code>node_modules</code>, coverage output, local <code>.env</code> files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.</p>



<p class="wp-block-paragraph">A <code>.dockerignore</code> file in the root of the context fixes it, and the rules apply to subdirectories too:</p>



<pre class="wp-block-code"><code>.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore</code></pre>



<p class="wp-block-paragraph">This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a <code>.git</code> directory or a stray credentials file in a production image, this is the line of defense that was missing.</p>



<p class="wp-block-paragraph">There&#8217;s a second option when the source is only needed to produce an artifact and shouldn&#8217;t live in the image at all. A bind mount makes the context available for the duration of one <code>RUN</code> without adding it to any layer:</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server</code></pre>



<p class="wp-block-paragraph">Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why <code>/app/server</code> sits outside <code>/build</code>. And mounted files are not persisted into the final image or the build cache. Only what the <code>RUN</code> produced survives.</p>



<h2 class="wp-block-heading">Failure family three: Dockerfile layer caching that never reaches CI</h2>



<p class="wp-block-paragraph">This is the invisible one, and it&#8217;s behind most &#8220;my Dockerfile is ordered correctly and CI is still slow&#8221; tickets.</p>



<p class="wp-block-paragraph">The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn&#8217;t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.</p>



<p class="wp-block-paragraph">The fix is an external cache. <code>--cache-to</code> exports the cache somewhere durable, <code>--cache-from</code> imports it at the start of the next build.</p>



<pre class="wp-block-code"><code>docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .</code></pre>



<p class="wp-block-paragraph">Four details decide whether this works or quietly does nothing:</p>



<ul class="wp-block-list">
<li><strong><code>mode=max</code> versus the default.</strong> The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. <code>mode=max</code> includes intermediate stages. It costs storage and upload time, and it is usually the right trade.</li>



<li><strong>Use a separate ref for the cache.</strong> A different tag or repository, anything that isn&#8217;t the tag you push the image to.</li>



<li><strong>Registry compatibility.</strong> The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting <code>image-manifest=true</code> alongside <code>oci-mediatypes=true</code> writes a single image manifest instead.</li>



<li><strong>The default driver is limited.</strong> The plain <code>docker</code> driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.</li>
</ul>



<p class="wp-block-paragraph">On GitHub Actions the equivalent lives in the build action:</p>



<pre class="wp-block-code"><code>- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max</code></pre>



<p class="wp-block-paragraph">The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you&#8217;d rather pay object storage rates than registry rates.</p>



<h2 class="wp-block-heading">Failure family four: cache mounts that don&#8217;t survive the job</h2>



<p class="wp-block-paragraph">Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt</code></pre>



<p class="wp-block-paragraph">Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:</p>



<pre class="wp-block-code"><code>RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update &amp;&amp; apt-get install --no-install-recommends -y gcc</code></pre>



<p class="wp-block-paragraph">Without <code>sharing=locked</code>, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.</p>



<p class="wp-block-paragraph">Now the part that isn&#8217;t in most tutorials. <strong>Cache mounts live on the builder&#8217;s local storage and they are not exported by <code>--cache-to</code>.</strong> The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.</p>



<p class="wp-block-paragraph">They&#8217;re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.</p>



<p class="wp-block-paragraph">One hard rule regardless: never put something the image needs into a cache mount. Installing into <code>/app/node_modules</code> behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager&#8217;s download directory, not its output.</p>



<h2 class="wp-block-heading">Failure family five: multi-stage builds and the COPY &#8211;link argument</h2>



<p class="wp-block-paragraph">Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]</code></pre>



<p class="wp-block-paragraph">The catch wastes an afternoon if you hit it cold: export cache in the default mode and the <code>deps</code> stage isn&#8217;t in the export, because it isn&#8217;t part of the final image. Multi-stage plus external cache means <code>mode=max</code>, or the whole structure buys you nothing in CI.</p>



<p class="wp-block-paragraph">Then there&#8217;s <code>COPY --link</code>, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn&#8217;t depend on the filesystem beneath it, then merges on top. The layer can be reused with <code>--cache-from</code> even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.</p>



<p class="wp-block-paragraph">The case against is also real, and it&#8217;s mostly semantics you didn&#8217;t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn&#8217;t match the announcement.</p>



<p class="wp-block-paragraph">Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn&#8217;t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don&#8217;t move, you&#8217;ve added a semantic footgun for nothing.</p>



<h2 class="wp-block-heading">Troubleshooting: proving which layer missed and why</h2>



<p class="wp-block-paragraph">Stop guessing. The build output tells you, if you ask for the plain progress format:</p>



<pre class="wp-block-code"><code>docker buildx build --progress=plain --tag app:dev .</code></pre>



<ol class="wp-block-list">
<li><strong>Find the first step not marked CACHED.</strong> Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.</li>



<li><strong>If it&#8217;s a COPY or ADD, the inputs changed.</strong> List what the context actually contains, compare against <code>.dockerignore</code>, and look for generated files or lockfiles being rewritten by a pre-build script.</li>



<li><strong>If it&#8217;s a RUN, something above it changed.</strong> Scroll up. A build arg, a label, a base image tag that moved, or <code>SOURCE_DATE_EPOCH</code> set from a commit timestamp.</li>



<li><strong>If the earliest steps miss in CI but not locally,</strong> the import didn&#8217;t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn&#8217;t a hard failure, the build just proceeds cold.</li>



<li><strong>If it&#8217;s intermittent, suspect eviction.</strong> Garbage collection or a cache storage budget removed what you were relying on.</li>
</ol>



<p class="wp-block-paragraph">Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:</p>



<pre class="wp-block-code"><code>docker buildx du
docker buildx prune --all</code></pre>



<p class="wp-block-paragraph">And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:</p>



<pre class="wp-block-code"><code>docker build --no-cache-filter deps --tag app:dev .</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Assuming the Dockerfile is the whole problem.</strong> If CI has no external cache configured, no amount of reordering helps. Check that first.</li>



<li><strong>Adding cache mounts to fix CI.</strong> Great locally, invisible in the pipeline unless the builder persists.</li>



<li><strong>Exporting cache in the default mode with a multi-stage build.</strong> The expensive stage isn&#8217;t in the export. This is the most common reason external caching looks configured and doesn&#8217;t help.</li>



<li><strong>Squashing everything into one giant RUN.</strong> Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.</li>



<li><strong>Treating a cached rebuild as a patched rebuild.</strong> A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.</li>



<li><strong>Using <code>latest</code> as a base tag.</strong> The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.</li>



<li>Write the <code>.dockerignore</code> before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.</li>



<li>Configure an external cache in CI on day one, with <code>mode=max</code> if the build has more than one stage.</li>



<li>Add cache mounts for every package manager in the build. One line each, and the difference between &#8220;added one dependency&#8221; and &#8220;re-downloaded the world&#8221;.</li>



<li>Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns &#8220;the build feels slow&#8221; into something you can point at.</li>



<li>Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why does my Docker build ignore the cache even though nothing changed?</h3>



<p class="wp-block-paragraph">Usually because something did change and it wasn&#8217;t the file you were watching. Build with <code>--progress=plain</code> and find the first step not marked CACHED. If it&#8217;s a COPY, a gap in <code>.dockerignore</code> is letting local artifacts into the context. If it&#8217;s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.</p>



<h3 class="wp-block-heading">Do fewer layers make Dockerfile layer caching better?</h3>



<p class="wp-block-paragraph">No, and the advice is often applied backwards. Combining commands into one <code>RUN</code> reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.</p>



<h3 class="wp-block-heading">Do cache mounts work in GitHub Actions?</h3>



<p class="wp-block-paragraph">Not usefully on the default hosted runners. Cache mounts live on the builder&#8217;s local storage and are not exported by <code>--cache-to</code>, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.</p>



<h3 class="wp-block-heading">Registry cache backend or the GitHub Actions one?</h3>



<p class="wp-block-paragraph">Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.</p>



<h3 class="wp-block-heading">Does changing a build secret invalidate the cache?</h3>



<p class="wp-block-paragraph">No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret&#8217;s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.</p>



<h3 class="wp-block-heading">Why does a cached image still contain old, vulnerable packages?</h3>



<p class="wp-block-paragraph">Because a <code>RUN</code> instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with <code>--no-cache</code>, target the relevant stage with <code>--no-cache-filter</code>, or move the base image tag so the layers below invalidate.</p>



<h3 class="wp-block-heading">Is COPY &#8211;link safe to use everywhere?</h3>



<p class="wp-block-paragraph">Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It&#8217;s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn&#8217;t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.</p>



<p class="wp-block-paragraph">So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn&#8217;t CACHED. If it&#8217;s step one, your problem isn&#8217;t the Dockerfile. It&#8217;s that the machine running the build has never seen your cache and nobody told it where to look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your build pipeline?</h2>



<p class="wp-block-paragraph">Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:</p>



<ul class="wp-block-list">
<li>Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache</li>



<li>Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases</li>



<li>Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI</li>



<li>Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small</li>



<li>Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early</li>



<li>Separating &#8220;rebuild for a code change&#8221; from &#8220;rebuild for a security patch&#8221; so cached layers stop hiding unpatched packages</li>
</ul>



<p class="wp-block-paragraph">Send me your Dockerfile and a build log with <code>--progress=plain</code> output, and I&#8217;ll tell you which step is costing you and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/dockerfile-layer-caching/">Dockerfile Layer Caching: Why Every Build Reinstalls Everything</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</title>
		<link>https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 10:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[GitLab CI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Jenkins]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=201</guid>

					<description><![CDATA[<p>A practical comparison of GitLab CI, GitHub Actions and Jenkins that skips the feature table. What actually decides the choice is where your code lives, whether the runner can reach the deploy target, and who owns the control plane at 2am. Includes the cost mechanics, the security failure modes, and a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question almost always arrives the same way. Someone pastes a long Jenkinsfile into a channel, adds a shrug emoji, and asks how hard it would be to move this to GitHub Actions. Nobody in the thread is asking the questions that will actually decide the outcome, because those questions are boring and the YAML is right there looking portable.</p>



<p class="wp-block-paragraph">Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.</p>



<p class="wp-block-paragraph">This post compares <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest &#8220;where it wins and where it doesn&#8217;t&#8221;, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.</p>



<h2 class="wp-block-heading">The part of the decision that actually bites</h2>



<p class="wp-block-paragraph">Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.</p>



<p class="wp-block-paragraph">Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.</p>



<p class="wp-block-paragraph">The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system &#8220;for now&#8221;, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.</p>



<p class="wp-block-paragraph">So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the &#8220;below the line&#8221; half cheapest for your specific situation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What each platform is really optimising for</h2>



<p class="wp-block-paragraph">Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.</p>



<h3 class="wp-block-heading">GitHub Actions: shortest path from push to green</h3>



<p class="wp-block-paragraph">Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else&#8217;s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.</p>



<p class="wp-block-paragraph">A minimal hardened job header looks like this, and every line in it is there for a reason:</p>



<pre class="wp-block-code"><code>name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"</code></pre>



<p class="wp-block-paragraph">The <code>permissions</code> block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The <code>concurrency</code> block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.</p>



<h3 class="wp-block-heading">GitLab CI: one platform owning the whole lifecycle</h3>



<p class="wp-block-paragraph">GitLab&#8217;s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.</p>



<p class="wp-block-paragraph">The pipeline language rewards you for using its graph features rather than treating stages as a queue:</p>



<pre class="wp-block-code"><code>workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh</code></pre>



<p class="wp-block-paragraph">The <code>workflow:rules</code> block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. <code>needs</code> turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. <code>interruptible: true</code> makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project&#8217;s CI settings; the keyword alone does nothing. And <code>expire_in</code> is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.</p>



<h3 class="wp-block-heading">Jenkins: whatever you need, on hardware you control</h3>



<p class="wp-block-paragraph">Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor&#8217;s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.</p>



<pre class="wp-block-code"><code>pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things in that file are worth pointing at. <code>timeout</code> and <code>buildDiscarder</code> are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And <code>timestamps()</code> and <code>cleanWs()</code> both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">GitLab CI vs GitHub Actions vs Jenkins on cost</h2>



<p class="wp-block-paragraph">Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.</p>



<ul class="wp-block-list"><li><strong>GitHub Actions.</strong> Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.</li><li><strong>GitLab CI.</strong> Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why &#8220;pay for seats, run your own runners&#8221; is such a common GitLab shape.</li><li><strong>Jenkins.</strong> No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.</li></ul>



<p class="wp-block-paragraph">Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.</p>



<p class="wp-block-paragraph">Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Security is where the differences stop being cosmetic</h2>



<p class="wp-block-paragraph">Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.</p>



<ul class="wp-block-list"><li><strong>Untrusted pull requests.</strong> This is the classic GitHub Actions trap. The <code>pull_request</code> trigger runs fork code without access to your secrets. The <code>pull_request_target</code> trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request&#8217;s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub&#8217;s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.</li><li><strong>Expression injection.</strong> Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.</li><li><strong>Self-hosted runners on public repositories.</strong> Don&#8217;t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.</li><li><strong>Mutable action and image references.</strong> Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.</li><li><strong>Credential scope.</strong> Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.</li><li><strong>Jenkins specifically.</strong> The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project&#8217;s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run in an afternoon</h2>



<ol class="wp-block-list"><li><strong>Follow platform gravity first.</strong> Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.</li><li><strong>Check whether a runner can reach the deploy target.</strong> Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.</li><li><strong>Name the on-call owner.</strong> Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.</li><li><strong>Model the compute, not the sticker price.</strong> Take last month&#8217;s build minutes, split them by operating system and machine size, and apply each platform&#8217;s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.</li><li><strong>Inventory what the plugins actually do.</strong> If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The &#8220;needs writing&#8221; pile is your real migration estimate.</li><li><strong>Trial with your ugliest pipeline.</strong> Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.</li></ol>



<h2 class="wp-block-heading">The hybrid pattern nobody puts on the slide</h2>



<p class="wp-block-paragraph">A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;Jenkins is legacy.&#8221;</strong> It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.</li><li><strong>&#8220;We&#8217;ll save money with self-hosted runners.&#8221;</strong> On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.</li><li><strong>&#8220;GitLab CI is just GitHub Actions with different YAML.&#8221;</strong> The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.</li><li><strong>&#8220;We&#8217;ll stay vendor-neutral by keeping all logic in shell scripts.&#8221;</strong> Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.</li><li><strong>&#8220;The bigger marketplace wins.&#8221;</strong> Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ul class="wp-block-list"><li>Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.</li><li>Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.</li><li>Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.</li><li>Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.</li><li>Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Jenkins still worth using?</h3>



<p class="wp-block-paragraph">Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.</p>



<h3 class="wp-block-heading">Which is cheaper, GitLab CI or GitHub Actions?</h3>



<p class="wp-block-paragraph">It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab&#8217;s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.</p>



<h3 class="wp-block-heading">Can I use GitLab CI with a GitHub repository?</h3>



<p class="wp-block-paragraph">There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.</p>



<h3 class="wp-block-heading">How long does migrating from Jenkins to GitHub Actions or GitLab CI take?</h3>



<p class="wp-block-paragraph">Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.</p>



<h3 class="wp-block-heading">Are self-hosted runners safe?</h3>



<p class="wp-block-paragraph">They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.</p>



<h3 class="wp-block-heading">Do I need to pin GitHub Actions to a commit SHA?</h3>



<p class="wp-block-paragraph">For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.</p>



<h3 class="wp-block-heading">Is it reasonable to run more than one CI system?</h3>



<p class="wp-block-paragraph">Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Comparing <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.</p>



<p class="wp-block-paragraph">Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.</p>



<h2 class="wp-block-heading">Need help choosing or moving a CI pipeline?</h2>



<p class="wp-block-paragraph">This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:</p>



<ul class="wp-block-list"><li>A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.</li><li>Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.</li><li>Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.</li><li>Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.</li><li>Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.</li><li>CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.</li><li>Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.</li></ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</title>
		<link>https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 11 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Health Checks]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=189</guid>

					<description><![CDATA[<p>Readiness probes are necessary for Kubernetes zero-downtime rollouts, but they only close one of four gaps. Here is what actually drops requests during a deploy: the race between SIGTERM and endpoint propagation, probes that lie, surge settings that quietly delete capacity, and load balancers that never watched EndpointSlices in the first place.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The rollout goes green. <code>kubectl rollout status</code> exits zero, every pod reports <code>1/1 Running</code>, and the deploy channel gets the thumbs-up emoji. Then someone from support pastes a screenshot: a spike of 502s, about forty seconds wide, lined up exactly with the deploy. Nobody paged. Nothing restarted. The graph just has a small cliff in it.</p>



<p class="wp-block-paragraph">That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that <strong>Kubernetes zero-downtime rollouts</strong> depend on, and it is not the question that usually bites.</p>



<p class="wp-block-paragraph">This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.</p>



<h2 class="wp-block-heading">What a zero-downtime rollout actually requires</h2>



<p class="wp-block-paragraph">Three independent things have to be true at the same time, and each is owned by a different part of the system.</p>



<ul class="wp-block-list">
<li><strong>Capacity never dips below what traffic needs.</strong> Owned by the Deployment&#8217;s rolling update strategy.</li>

<li><strong>No pod receives traffic before it can serve it.</strong> Owned by the readiness probe.</li>

<li><strong>No pod receives traffic after it stops serving it.</strong> Owned by graceful termination, and this is the one nobody configures.</li>
</ul>



<p class="wp-block-paragraph">Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.</p>



<h2 class="wp-block-heading">Failure family one: traffic arrives before the pod can serve it</h2>



<p class="wp-block-paragraph">This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means &#8220;the process was forked&#8221;, not &#8220;the process can answer HTTP&#8221;. The pod lands in the Service&#8217;s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.</p>



<h3 class="wp-block-heading">The readiness probe that lies</h3>



<p class="wp-block-paragraph">Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that returns 200 from a static handler.</strong> If <code>/healthz</code> is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that checks downstream dependencies.</strong> This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks <em>every replica</em> unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers &#8220;can this pod serve traffic&#8221;, not &#8220;is the whole system healthy&#8221;. Check the dependency in your app&#8217;s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.</p>



<h3 class="wp-block-heading">Let the startup probe own the boot budget</h3>



<p class="wp-block-paragraph">For anything slow to boot, do not stretch <code>initialDelaySeconds</code> on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.</p>



<pre class="wp-block-code"><code># The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3</code></pre>



<p class="wp-block-paragraph">Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: traffic arrives after the pod has stopped</h2>



<p class="wp-block-paragraph">This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has &#8220;already added readiness probes&#8221;.</p>



<p class="wp-block-paragraph">When a pod is deleted, two things happen <em>in parallel</em>, not in sequence:</p>



<ol class="wp-block-list">
<li>The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container&#8217;s main process.</li>

<li>The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.</li>
</ol>



<p class="wp-block-paragraph">Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.</p>



<h3 class="wp-block-heading">The preStop sleep, and what it does not do</h3>



<p class="wp-block-paragraph">The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.</p>



<pre class="wp-block-code"><code>spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15</code></pre>



<p class="wp-block-paragraph">If you are on an older cluster without the native <code>sleep</code> handler, the equivalent is an exec hook, which does require a shell and a <code>sleep</code> binary in the image:</p>



<pre class="wp-block-code"><code>lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]</code></pre>



<p class="wp-block-paragraph">Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application&#8217;s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.</p>



<p class="wp-block-paragraph">There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.</p>



<h3 class="wp-block-heading">The grace period is a shared budget</h3>



<p class="wp-block-paragraph">This trips people up. The <code>terminationGracePeriodSeconds</code> countdown starts when the pod is marked Terminating, which is <em>before</em> the preStop hook runs, not after. The hook and your application&#8217;s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.</p>



<p class="wp-block-paragraph">Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.</p>



<p class="wp-block-paragraph">One more detail worth knowing before you go looking for it: probes accept their own <code>terminationGracePeriodSeconds</code> override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the rollout deletes capacity faster than it adds it</h2>



<p class="wp-block-paragraph">The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.</p>



<pre class="wp-block-code"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1</code></pre>



<p class="wp-block-paragraph">Setting <code>maxUnavailable: 0</code> is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise <code>maxSurge</code> to a percentage if you have the node capacity and want the speed back.</p>



<p class="wp-block-paragraph"><code>minReadySeconds</code> is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.</p>



<p class="wp-block-paragraph">Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.</p>



<pre class="wp-block-code"><code>apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api</code></pre>



<h2 class="wp-block-heading">Failure family four: the load balancer was never watching EndpointSlices</h2>



<p class="wp-block-paragraph">Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.</p>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.</p>



<p class="wp-block-paragraph">Two things help here:</p>



<ul class="wp-block-list">
<li><strong>Pod readiness gates.</strong> These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.</li>

<li><strong>Match your preStop sleep to the deregistration delay.</strong> If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.</li>
</ul>



<p class="wp-block-paragraph">The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod&#8217;s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.</p>



<h2 class="wp-block-heading">Verifying a zero-downtime rollout instead of hoping</h2>



<p class="wp-block-paragraph">A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.</p>



<ol class="wp-block-list">
<li><strong>Generate steady traffic against the real ingress path.</strong> Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.</li>

<li><strong>Trigger a rollout with no image change</strong>, so you are testing the mechanism rather than your new code.</li>

<li><strong>Watch EndpointSlices in a second terminal</strong> and note how long a terminating pod stays in the list.</li>

<li><strong>Count non-200 responses.</strong> Anything above zero is a bug, not noise.</li>

<li><strong>Repeat under realistic concurrency.</strong> A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.</li>
</ol>



<pre class="wp-block-code"><code># Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m</code></pre>



<p class="wp-block-paragraph">Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Rollout hangs, new pods never become Ready.</strong> The readiness probe is failing. <code>kubectl describe pod</code> shows the probe failure and the response it got. Check that the port name in the probe matches a declared <code>containerPort</code> name, that the path exists, and that the app is listening on all interfaces rather than only <code>127.0.0.1</code>, which is a classic one when moving from a local docker-compose setup.</p>



<p class="wp-block-paragraph"><strong>Probes fail only under load.</strong> Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within <code>timeoutSeconds</code>, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.</p>



<p class="wp-block-paragraph"><strong>Errors persist after adding a preStop sleep.</strong> Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.</p>



<p class="wp-block-paragraph"><strong>Pods stuck Terminating for the full grace period.</strong> Your app is ignoring SIGTERM. This is extremely common when the container&#8217;s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of <code>ENTRYPOINT</code>, or an init like <code>tini</code>, so your process actually receives the signal.</p>



<p class="wp-block-paragraph"><strong>Rollout succeeded but the new version is broken.</strong> Roll back first, investigate second.</p>



<pre class="wp-block-code"><code># Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.</li>

<li>Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.</li>

<li>Leaving <code>maxUnavailable</code> at the default on a service running near capacity.</li>

<li>Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.</li>

<li>Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.</li>

<li>Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.</li>

<li>Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Three separate endpoints: <code>/healthz/started</code>, <code>/healthz/ready</code>, <code>/healthz/live</code>. They answer different questions and should be allowed to disagree.</li>

<li><code>maxUnavailable: 0</code> plus a <code>maxSurge</code> you have node capacity for, as the default for anything user-facing.</li>

<li>A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.</li>

<li>A grace period that covers preStop plus your slowest realistic request, with slack.</li>

<li>Handle SIGTERM properly in the application, and make sure it reaches PID 1.</li>

<li>A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.</li>

<li><code>minReadySeconds</code> long enough to catch a pod that passes its probe and then dies.</li>

<li>Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do readiness probes alone give me zero-downtime deployments?</h3>



<p class="wp-block-paragraph">No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.</p>



<h3 class="wp-block-heading">How long should the preStop sleep be?</h3>



<p class="wp-block-paragraph">Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.</p>



<h3 class="wp-block-heading">Should the readiness probe check the database?</h3>



<p class="wp-block-paragraph">Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.</p>



<h3 class="wp-block-heading">What is the difference between maxSurge and maxUnavailable?</h3>



<p class="wp-block-paragraph"><code>maxSurge</code> is how many pods you may run <em>above</em> the replica count during a rollout. <code>maxUnavailable</code> is how many you may drop <em>below</em> it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.</p>



<h3 class="wp-block-heading">Why do I still get 502s on EKS, GKE or another managed cluster?</h3>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group&#8217;s deregistration delay rather than against kube-proxy.</p>



<h3 class="wp-block-heading">Can I get zero-downtime rollouts with a StatefulSet?</h3>



<p class="wp-block-paragraph">The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.</p>



<h3 class="wp-block-heading">Does kubectl rollout status prove the deploy was clean?</h3>



<p class="wp-block-paragraph">It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.</p>



<p class="wp-block-paragraph">Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your rollouts?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change</li>

<li>Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number</li>

<li>Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager</li>

<li>Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths</li>

<li>Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks</li>

<li>Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event</li>
</ul>



<p class="wp-block-paragraph">If you have a Deployment manifest, a <code>kubectl describe pod</code> output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</title>
		<link>https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 10 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AppSync]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Amplify]]></category>
		<category><![CDATA[BaaS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Firebase]]></category>
		<category><![CDATA[Firestore]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=177</guid>

					<description><![CDATA[<p>Both platforms look identical on day one: a schema file, a typed client, a deploy that works. The difference shows up months later, in the queries you cannot afford and the layer you cannot leave. A practical comparison of data models, billing shape, auth stickiness and exit cost.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/">AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that ends the honeymoon usually looks harmless. Something like: <em>show me last quarter&#8217;s orders, grouped by region, filtered to three statuses, sorted by value</em>. Someone picks it up, opens the backend, and works out that there is no way to answer it without either pulling the whole collection down to the client or standing up a second database next to the first one.</p>



<p class="wp-block-paragraph">That is the moment the <strong>AWS Amplify vs Firebase</strong> decision actually gets made. Not on day one, when both platforms look like a schema file, a typed client and a deploy that just works. On day two hundred, when the read path you committed to without noticing starts charging rent.</p>



<p class="wp-block-paragraph">This post is a comparison, but not a feature tour. Both vendors publish good feature tables and neither of them is lying. What they do not tell you is which parts of the platform you can walk away from cheaply and which parts you are stuck with. So this is organised around the four things that decide that: the data model, the billing shape, the auth layer, and the exit cost.</p>



<h2 class="wp-block-heading">The failure mode that bites: your read path is decided on day one</h2>



<p class="wp-block-paragraph">Both platforms hand you a pleasant abstraction over a NoSQL store. Firebase gives you Firestore, a document database. Amplify&#8217;s data layer, by default, gives you AppSync sitting in front of DynamoDB. In both cases you write a schema, get a typed client, and start shipping.</p>



<p class="wp-block-paragraph">Here is what neither schema file makes obvious: it is not a description of your data. It is a bet on the queries you will need. Change the bet later and you are not editing a schema, you are backfilling.</p>



<p class="wp-block-paragraph">In Firestore, every query shape needs an index that supports it. Filter on two fields and sort on a third and you need a composite index. The console will helpfully offer to create it for you, which is why most teams never notice they are accumulating a per-query-shape index budget. There are no joins. If an order needs its customer&#8217;s name, either you denormalise the name into the order document or you do a second fetch per row.</p>



<p class="wp-block-paragraph">In Amplify, relationships between models are resolved by AppSync, which usually means an extra round trip per relationship. Secondary indexes have to be declared in the schema so the underlying table can be built to serve them. Same constraint, different vocabulary.</p>



<p class="wp-block-paragraph">The good news is that both vendors have finally admitted this is a problem. Firebase Data Connect puts a managed Cloud SQL for PostgreSQL instance behind a GraphQL schema and generates typed SDKs against it. On the AWS side, Amplify&#8217;s data layer can be pointed at a SQL database rather than DynamoDB, and the newer Blocks approach lets you compose Postgres and other capabilities into an existing Amplify backend. If your application has anything resembling reporting, joins or ad-hoc filtering in its future, start there rather than starting on documents and migrating under pressure.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">AWS Amplify: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Amplify Gen 2 defines the whole backend in TypeScript. Auth, data, storage and functions all live as code in an <code>amplify/</code> directory, get reviewed in pull requests, and compile down through CDK to CloudFormation.</p>



<pre class="wp-block-code"><code>// amplify/data/resource.ts
import { type ClientSchema, a, defineData } from '@aws-amplify/backend';

const schema = a.schema({
  Order: a
    .model({
      region: a.string().required(),
      status: a.enum(['PENDING', 'SHIPPED', 'CANCELLED']),
      total: a.float(),
    })
    .authorization((allow) =&gt; [allow.owner()]),
});

export type Schema = ClientSchema&lt;typeof schema&gt;;

export const data = defineData({ schema });</code></pre>



<p class="wp-block-paragraph">Two things in that snippet matter more than they look. The <code>authorization</code> call is not a comment about intent, it generates the actual resolver-level access rules, so an access change is a code change with a diff. And <code>ClientSchema</code> flows the backend types into the frontend, which means removing a field breaks the build rather than breaking production at 3am.</p>



<p class="wp-block-paragraph">Development happens in a per-developer cloud sandbox. It watches the <code>amplify/</code> folder and redeploys on save:</p>



<pre class="wp-block-code"><code># Deploy and watch your own isolated backend stack
npx ampx sandbox

# Tear it down when you're finished - it is real infrastructure
npx ampx sandbox delete</code></pre>



<h3 class="wp-block-heading">Where Amplify wins</h3>



<ul class="wp-block-list">
<li><strong>You are already on AWS.</strong> If your organisation has accounts, IAM boundaries, a VPC design and a compliance story, Amplify slots into all of it. Firebase would mean a second cloud to audit.</li>

<li><strong>The escape hatch is real.</strong> Because the backend compiles to CDK, you can drop into raw constructs and attach any AWS service without leaving the framework. That ceiling is much higher than Firebase&#8217;s.</li>

<li><strong>Environments are branches.</strong> Git branch to deployed environment is the native model, so staging is not a thing you build, it is a thing you push.</li>

<li><strong>Type safety end to end.</strong> This genuinely catches a class of bug that Firebase projects tend to find in production.</li>
</ul>



<h3 class="wp-block-heading">Where Amplify doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Gen 1 is on a clock.</strong> Amplify Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. This matters because a lot of the tutorials, Stack Overflow answers and agency codebases you will encounter are Gen 1, and Gen 1 to Gen 2 is a migration, not an upgrade. AWS ships tooling for it, but the guide describes a blue/green run with an irreversible refactor step. Budget for it properly.</li>

<li><strong>&#8220;Local&#8221; development is not local.</strong> The sandbox is a real CloudFormation stack in a real account. It is high fidelity, which is the point, but your inner loop now includes deploy time and your dev environment costs money.</li>

<li><strong>Errors surface as AWS errors.</strong> A mistake in a TypeScript schema can come back as a CloudFormation rollback referencing a resource you never named. Debugging Amplify means being comfortable reading AppSync, DynamoDB and CloudWatch.</li>

<li><strong>Smaller community.</strong> Fewer answers exist, and a meaningful share of the ones that do exist are for the previous generation.</li>
</ul>



<h2 class="wp-block-heading">Google Firebase: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Firebase&#8217;s advantage is not that it is simpler. It is that the whole loop, from writing a rule to seeing a client behave, is shorter. The emulator suite runs the pieces on your machine:</p>



<pre class="wp-block-code"><code># Run auth, Firestore and functions locally, no cloud project touched
firebase emulators:start --only auth,firestore,functions</code></pre>



<p class="wp-block-paragraph">The thing you will spend the most time on is security rules, and it is worth understanding why they are shaped the way they are:</p>



<pre class="wp-block-code"><code>rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /orders/{orderId} {
      allow read: if request.auth != null
                  &amp;&amp; request.auth.uid == resource.data.ownerUid;

      allow create: if request.auth != null
                    &amp;&amp; request.auth.uid == request.resource.data.ownerUid;

      allow update, delete: if false;
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The distinction people get wrong is <code>resource.data</code> versus <code>request.resource.data</code>. The first is the document as it exists in the database, so it protects the read path. The second is the document the client is trying to write, so it protects the write path. Use the wrong one in a create rule and you are checking ownership on a document that does not exist yet, which evaluates to nothing and quietly lets the write through. That is the kind of bug that ships.</p>



<h3 class="wp-block-heading">Where Firebase wins</h3>



<ul class="wp-block-list">
<li><strong>Mobile.</strong> Crashlytics, Cloud Messaging, Remote Config, A/B testing and analytics are one SDK away and genuinely good. AWS has equivalents, but not assembled like this.</li>

<li><strong>Offline and realtime.</strong> Firestore&#8217;s offline persistence and listener model handle flaky connectivity without you designing for it. If your users are on trains or in warehouses, this is a serious argument.</li>

<li><strong>Emulators.</strong> Being able to run the backend on a laptop, in CI, with no cloud project, is a real productivity difference against Amplify&#8217;s cloud sandbox.</li>

<li><strong>Time to first screen.</strong> For a prototype or an MVP that needs to exist by Friday, Firebase usually wins on speed alone.</li>
</ul>



<h3 class="wp-block-heading">Where Firebase doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Configuration drifts out of git.</strong> The console is convenient and it is also a second source of truth. Indexes get created from an error link, a provider gets enabled by hand, and six months later nobody can rebuild the project from the repository. Terraform and the Firebase CLI both help, but you have to decide to use them.</li>

<li><strong>Security rules are a language, not a checkbox.</strong> They are also evaluated on the server for every access, which has cost implications when rules perform document lookups.</li>

<li><strong>Features you assume are included are an upgrade.</strong> Multi-factor auth, SAML and OIDC, blocking functions, audit logging and multi-tenancy live behind the Identity Platform upgrade, which comes with a different pricing model. On the free Spark plan, upgrading also introduces a daily active user cap. Find this out during design, not during an enterprise sales cycle.</li>

<li><strong>The ceiling is lower.</strong> When you outgrow Firebase you are usually moving to Google Cloud proper, which is a different set of tools and a different mental model.</li>
</ul>



<h2 class="wp-block-heading">What actually generates the bill</h2>



<p class="wp-block-paragraph">Rates change, so treat published numbers as current-at-time-of-reading and go to the vendor pricing pages for the arithmetic. What is stable, and what you should actually reason about, is the <em>shape</em> of the meters.</p>



<p class="wp-block-paragraph">Firestore charges per document, and the details are where teams get caught:</p>



<ul class="wp-block-list">
<li>A query that returns 200 documents is 200 reads, whether you use two fields from each one or forty.</li>

<li>A realtime listener is charged a read each time a document in its result set is added or updated. Attach a listener to a busy collection and you have built a meter that runs while nobody is looking at the screen.</li>

<li>Where your SDK exposes query offsets, skipped documents are still billed. Paginate with cursors, not offsets.</li>

<li>Aggregations such as <code>count()</code> are billed per batch of index entries read, with a minimum of one document read. Cheap, but not free, and not zero when you run them per page load.</li>

<li>Stored bytes include indexes and metadata, so index sprawl shows up twice: once in write amplification, once in storage.</li>
</ul>



<p class="wp-block-paragraph">Amplify has no single meter, because Amplify is not a service. The bill is assembled from AppSync (query and mutation operations, realtime update messages, and connection-minutes for open WebSockets), DynamoDB, Lambda invocations and duration, CloudWatch log ingestion and retention, Cognito monthly active users, and Amplify Hosting build minutes plus data served.</p>



<p class="wp-block-paragraph">That difference is the practical one. Firebase gives you a bill you can read in the console in thirty seconds. AWS gives you eight line items that will not tell you which feature caused them unless you tagged resources properly on the way in. If you go the AWS route, set up cost allocation tags and a budget alarm before launch, not after the first invoice. Pushing CloudWatch metrics into <a href="https://grafana.com/" target="_blank" rel="noreferrer noopener">Grafana</a> alongside your application metrics is worth the afternoon it takes, because cost anomalies and traffic anomalies are usually the same incident viewed twice.</p>



<h2 class="wp-block-heading">Auth is the component you cannot cheaply replace</h2>



<p class="wp-block-paragraph">Firebase Authentication and Amazon Cognito are both fine. Neither is the reason to pick a platform. But auth is the layer where switching hurts most, because you are not migrating data, you are migrating <em>sessions and identities</em>, and every user notices.</p>



<p class="wp-block-paragraph">Both let you export users, including password hashes, so a migration is technically possible. What is hard is everything hanging off the user ID: rules that compare against it, resolver authorisation that assumes it, third-party integrations keyed on it, and support tooling that looks users up by it.</p>



<p class="wp-block-paragraph">One habit removes most of that pain regardless of which platform you pick: store your own internal user record from the first commit, with your own primary key, and treat the provider&#8217;s UID as an external identifier on that record. It costs you one table and one lookup. It converts an auth migration from a rewrite into a mapping exercise.</p>



<h2 class="wp-block-heading">Hosting is the layer nobody should agonise over</h2>



<p class="wp-block-paragraph">Amplify Hosting deploys from a git branch and supports server-rendered frameworks. Firebase now has two: the original Hosting for static sites, and App Hosting for modern server-rendered apps, which is generally available and runs on Cloud Run with Cloud CDN in front.</p>



<p class="wp-block-paragraph">Both are good. Neither should influence your decision much, because hosting is the one layer with genuine competition and near-zero switching cost. A Next.js or Nuxt app moves to Vercel, Netlify, Cloudflare Pages or a container on a VPS from a provider like <a href="https://www.interserver.net/" target="_blank" rel="noreferrer noopener">InterServer</a> or DigitalOcean in an afternoon. Put a CDN such as <a href="https://www.cloudflare.com/" target="_blank" rel="noreferrer noopener">Cloudflare</a> in front and the origin becomes an implementation detail your users never see.</p>



<p class="wp-block-paragraph">Spend your deliberation budget on the data layer instead. That is where it pays.</p>



<h2 class="wp-block-heading">Exit cost, ranked</h2>



<p class="wp-block-paragraph">If you rank the layers by what it costs to leave, the ordering is the same on both platforms and it is the opposite of the ordering people use when choosing:</p>



<ol class="wp-block-list">
<li><strong>Hosting.</strong> Swap it in a day. Barely counts as lock-in.</li>

<li><strong>Functions.</strong> Cloud Functions and Lambda handlers are mostly your own code with a different signature wrapped around it. A rewrite, but a bounded one.</li>

<li><strong>Auth.</strong> A user migration project with a communications plan attached.</li>

<li><strong>The data model.</strong> You do not migrate this. You rebuild the read path. Firestore&#8217;s managed export produces Firestore-format files in Cloud Storage, and DynamoDB exports to S3, and in both cases what lands is denormalised documents shaped by decisions you made two years ago, not a relational dataset you can point a reporting tool at.</li>
</ol>



<p class="wp-block-paragraph">There is a fifth item with no clean export at all: realtime. Firestore listeners and AppSync subscriptions have no portable equivalent, so if realtime is core to your product, whichever you choose is close to permanent. That is not automatically an argument against either. It is an argument for knowing you are making that commitment when you make it.</p>



<h2 class="wp-block-heading">AWS Amplify vs Firebase: how I&#8217;d actually decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A procedure, in order, stopping at the first clear answer:</p>



<ol class="wp-block-list">
<li><strong>Write down your three hardest read queries.</strong> Not the CRUD. The reporting screen, the search, the admin filter. If you cannot write them, you are not ready to choose a backend.</li>

<li><strong>Do any of them need joins, aggregates or filters you cannot predict?</strong> If yes, start relational: Firebase Data Connect, Amplify over a SQL data source, or plain Postgres behind an API you own. Do not start on documents planning to fix it later.</li>

<li><strong>Is this mobile-first with real offline requirements?</strong> Firebase, and it is not close.</li>

<li><strong>Does your organisation already run on AWS?</strong> Existing accounts, IAM, VPCs, audit posture and a security team who will ask questions about a second cloud all point to Amplify, even if Firebase would be faster this quarter.</li>

<li><strong>Who is maintaining this in two years?</strong> Amplify Gen 2 expects TypeScript fluency and some tolerance for CDK and CloudFormation. Firebase expects less. Match the platform to the team you will actually have.</li>

<li><strong>Any data residency or regulatory constraints?</strong> Check region availability for every service you plan to use, not just the headline one, before you commit.</li>

<li><strong>Still tied?</strong> It is a prototype. Pick the one your team can ship this week, and write down explicitly that the data model is the part you will have to revisit.</li>
</ol>



<h2 class="wp-block-heading">Mistakes I see repeatedly</h2>



<ul class="wp-block-list">
<li><strong>Choosing on SDK ergonomics.</strong> Both SDKs are pleasant. You are picking a data model and a billing model, and the SDK is the layer you would replace most easily.</li>

<li><strong>Building reporting on the transactional store.</strong> Both platforms punish this. Stream to a warehouse or a read replica and let the operational store do one job.</li>

<li><strong>Launching without a budget alarm.</strong> A missing <code>limit()</code>, a listener on a busy collection, or one post that gets traction can turn a hobby bill into a real one overnight. Set the alarm before you need it.</li>

<li><strong>Treating authorisation as a later task.</strong> On both platforms, access control is coupled to the data model. Retrofitting it usually means reshaping documents.</li>

<li><strong>Following Amplify Gen 1 material.</strong> The commands, the directives and the mental model are all different. Check which generation a tutorial targets before you follow it.</li>

<li><strong>Letting console clicks become infrastructure.</strong> Anything you enabled by hand is something you cannot rebuild. Get it into the CLI config or into Terraform.</li>

<li><strong>Assuming the free tier is the plan.</strong> It is a trial of the plan. Model your costs at ten times current traffic and see whether you still like the answer.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS Amplify or Firebase cheaper?</h3>



<p class="wp-block-paragraph">It depends entirely on your access patterns, and anyone who answers this without seeing them is guessing. Firebase tends to be cheaper and far more predictable at small scale, partly because the free allowances are generous and partly because there is one meter to watch. AWS tends to win when you have steady, high-volume traffic and the engineering discipline to tune DynamoDB capacity and Lambda sizing. The bigger practical difference is legibility: a Firebase bill tells you what happened, an AWS bill tells you what was consumed.</p>



<h3 class="wp-block-heading">Can I move from Firebase to AWS Amplify later?</h3>



<p class="wp-block-paragraph">Yes, but price it by layer rather than as one project. Hosting and functions move cheaply. Auth is a real migration with user impact. The data model is usually a rebuild rather than a move, because a Firestore export gives you denormalised documents, not a relational dataset. Teams that manage this well are the ones that kept business logic out of security rules and out of client code from the beginning.</p>



<h3 class="wp-block-heading">Is Amplify Gen 1 still supported?</h3>



<p class="wp-block-paragraph">Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. It receives critical fixes and security patches, not new features. If you are starting something new, use Gen 2. If you have a Gen 1 application in production, schedule the migration deliberately using the official tooling and guide, and test thoroughly before the refactor step, which is difficult to reverse.</p>



<h3 class="wp-block-heading">Do I need Firebase Data Connect, or is Firestore enough?</h3>



<p class="wp-block-paragraph">Firestore is enough when your reads are predictable, mostly key-based, and denormalise cleanly. The moment you need joins, aggregates across collections, or filters that product will keep changing, Data Connect&#8217;s managed Postgres is the better foundation. It is also a lot easier to adopt at the start than to migrate onto after a year of document design.</p>



<h3 class="wp-block-heading">Which one is better for a mobile app?</h3>



<p class="wp-block-paragraph">Firebase, in most cases. The offline persistence model, the crash reporting, the push messaging and the remote configuration are all mature and designed to work together. Amplify has mobile libraries and they work, but you are assembling the surrounding pieces from separate AWS services rather than getting them as a set.</p>



<h3 class="wp-block-heading">What about a self-hosted alternative?</h3>



<p class="wp-block-paragraph">Worth considering if lock-in is your main worry. Supabase and Appwrite both offer a comparable developer experience over Postgres, and either can run on your own infrastructure. Understand what you are taking on, though: backups, restore testing, upgrades, patching and monitoring all become yours. That is a real operational load, and it is the trade most teams are implicitly paying a managed vendor to avoid.</p>



<h3 class="wp-block-heading">Can I use both together?</h3>



<p class="wp-block-paragraph">You can, and some teams do run Firebase for analytics and messaging alongside an AWS backend. Be honest about the cost: two identity models, two billing accounts, two audit trails and two on-call runbooks. Do it because one platform is clearly better at a specific job, not because it was easier than making a decision.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Amplify vs Firebase</strong> question is almost never settled by features, and it is never settled by which SDK feels nicer in a tutorial. Both platforms will get you to a working application quickly, and both will hold up in production for the workloads they were designed for.</p>



<p class="wp-block-paragraph">What separates them, months later, is the read path you committed to on day one and the layer you cannot afford to leave. Write down your three hardest queries before you write any schema. Decide whether you need relational storage while it is still a decision rather than a migration. Keep your own user record from the first commit. Set a budget alarm before launch.</p>



<p class="wp-block-paragraph">Do those four things and either platform is a reasonable choice. Skip them and neither one saves you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion before you commit?</h2>



<p class="wp-block-paragraph">Backend platform decisions are cheap to review and expensive to reverse. Things I help with in this area:</p>



<ul class="wp-block-list">
<li>Reviewing a Firestore or DynamoDB data model against the queries you actually need, before the design hardens</li>

<li>Cost modelling both platforms against your real traffic assumptions, with the meters broken out so you can see what drives the bill</li>

<li>Amplify Gen 1 to Gen 2 migration planning, including the blue/green sequence and what to verify before the irreversible step</li>

<li>Auditing Firestore security rules and Amplify authorisation rules for gaps between the read path and the write path</li>

<li>Setting up cost allocation tags, budget alarms and CloudWatch or Grafana dashboards so a runaway query shows up in hours rather than on the invoice</li>

<li>Exit planning: working out what a migration off your current platform would genuinely cost, layer by layer</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete answer rather than a general one, send me the thing itself: your schema file, your security rules, a screenshot of last month&#8217;s billing breakdown, or the three queries you are worried about. That is usually enough to tell you something useful.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/">AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Docker Compose in Production: What Works and What Quietly Burns You</title>
		<link>https://john-nessime.com/blog/devops/docker-compose-in-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 07 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[Health Checks]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reverse Proxy]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Volumes]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=159</guid>

					<description><![CDATA[<p>Running Docker Compose in production is a reasonable choice for a single host, but the defaults were chosen for a laptop. A walk through the failure families that actually bite: the deploy gap, unrotated logs filling the disk, anonymous volumes, published ports that bypass your firewall, secrets in environment variables, and health checks that report without acting.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-compose-in-production/">Docker Compose in Production: What Works and What Quietly Burns You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The stack had been up for months. Nobody had touched it, nobody had needed to. Then the disk alert fires at an awkward hour, Postgres flips to read-only because it cannot write WAL, and you SSH in to find that one chatty container has written tens of gigabytes into a single JSON log file that nothing was ever going to rotate.</p>



<p class="wp-block-paragraph">That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn&#8217;t.</p>



<p class="wp-block-paragraph">Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.</p>



<p class="wp-block-paragraph">This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.</p>



<h2 class="wp-block-heading">What Docker Compose genuinely gets right</h2>



<p class="wp-block-paragraph">Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.</p>



<ul class="wp-block-list">
<li><strong>The whole system fits in one file you can read.</strong> Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.</li>

<li><strong>Dev and prod can share a base file.</strong> Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.</li>

<li><strong>No control plane to operate.</strong> A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.</li>

<li><strong>Recovery is comprehensible.</strong> When something breaks at 2am, <code>docker compose ps</code> and <code>docker compose logs</code> tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.</li>
</ul>



<p class="wp-block-paragraph">If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.</p>



<h2 class="wp-block-heading">Failure family one: the deploy gap</h2>



<p class="wp-block-paragraph">This is the one that surprises people who came from a platform that did rolling updates for them.</p>



<p class="wp-block-paragraph"><code>docker compose up -d</code> is not a rolling update. When a service&#8217;s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.</p>



<p class="wp-block-paragraph">The <code>deploy.update_config</code> block with <code>order: start-first</code> exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<p class="wp-block-paragraph">Three options, in increasing order of effort:</p>



<ol class="wp-block-list">
<li><strong>Accept the gap and shrink it.</strong> Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.</li>

<li><strong>Put a reverse proxy in front and run two slots.</strong> Traefik, Caddy or plain Nginx in a container, with <code>app-blue</code> and <code>app-green</code> services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.</li>

<li><strong>Drain at the edge.</strong> If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:</p>



<pre class="wp-block-code"><code># Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.</code></pre>



<p class="wp-block-paragraph">That last line matters more than it looks. A service with no health check is invisible to <code>--wait</code>, so a deploy script can report success while your API is still crash-looping.</p>



<h2 class="wp-block-heading">Failure family two: the disk fills up and nothing tells you</h2>



<p class="wp-block-paragraph">Docker&#8217;s default logging driver is <code>json-file</code>, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the <code>local</code> driver is the recommended alternative because it rotates out of the box and uses a more compact format.</p>



<p class="wp-block-paragraph">So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.</p>



<p class="wp-block-paragraph">Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:</p>



<pre class="wp-block-code"><code>// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.</p>



<p class="wp-block-paragraph">You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:</p>



<pre class="wp-block-code"><code>x-logging: &amp;default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    &lt;&lt;: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    &lt;&lt;: *default-logging</code></pre>



<p class="wp-block-paragraph">Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.</p>



<pre class="wp-block-code"><code># Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v</code></pre>



<h2 class="wp-block-heading">Failure family three: state you did not mean to keep</h2>



<p class="wp-block-paragraph">Two related traps here.</p>



<p class="wp-block-paragraph">The first is anonymous volumes. If an image declares a <code>VOLUME</code> in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.</p>



<p class="wp-block-paragraph">The second is <code>docker compose down</code>. On its own it removes containers and the project&#8217;s networks but leaves named volumes alone, which is the behaviour you want. Add <code>-v</code> and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.</p>



<p class="wp-block-paragraph">The practical rule: on a production host, restarting services is <code>docker compose up -d</code> or <code>docker compose restart</code>. <code>down</code> belongs in a maintenance procedure with a fresh backup, not in a daily habit.</p>



<p class="wp-block-paragraph">Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.</p>



<h2 class="wp-block-heading">Failure family four: ports that bypass your firewall</h2>



<p class="wp-block-paragraph">This is the one that turns into a security incident rather than an outage.</p>



<p class="wp-block-paragraph">When you write <code>ports: - "5432:5432"</code>, Docker publishes that port on all host interfaces and inserts its own rules into the kernel&#8217;s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a <code>ufw status</code> output that looks perfectly reassuring.</p>



<p class="wp-block-paragraph">The fix is to stop publishing what does not need publishing:</p>



<pre class="wp-block-code"><code>services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]</code></pre>



<p class="wp-block-paragraph">Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.</p>



<p class="wp-block-paragraph">While you are in daemon config: set <code>default-address-pools</code> as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.</p>



<h2 class="wp-block-heading">Failure family five: secrets in environment variables</h2>



<p class="wp-block-paragraph">Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in <code>docker inspect</code>, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.</p>



<p class="wp-block-paragraph">The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under <code>/run/secrets/</code>:</p>



<pre class="wp-block-code"><code>services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt</code></pre>



<p class="wp-block-paragraph">The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container&#8217;s environment, not in <code>docker inspect</code> output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.</p>



<p class="wp-block-paragraph">One more thing that trips people up: <code>.env</code> and <code>env_file</code> are different mechanisms. The <code>.env</code> file in the project directory feeds variable interpolation <em>inside</em> the Compose file. <code>env_file</code> passes variables <em>into</em> the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.</p>



<h2 class="wp-block-heading">Failure family six: health checks that report but never act</h2>



<p class="wp-block-paragraph">A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.</p>



<p class="wp-block-paragraph">So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.</p>



<p class="wp-block-paragraph">Health checks are still worth defining, because they gate startup ordering and they gate <code>--wait</code>. Just be clear about what they do not do.</p>



<pre class="wp-block-code"><code>services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s</code></pre>



<p class="wp-block-paragraph">Note <code>restart: unless-stopped</code> rather than <code>always</code>. The difference shows up after a host reboot: <code>always</code> will start a container you had deliberately stopped, <code>unless-stopped</code> respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.</p>



<p class="wp-block-paragraph">To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.</p>



<h2 class="wp-block-heading">Troubleshooting a Compose stack that is misbehaving</h2>



<p class="wp-block-paragraph">The commands I reach for, roughly in order:</p>



<pre class="wp-block-code"><code># The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' &lt;container&gt;

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"</code></pre>



<p class="wp-block-paragraph">Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Deploying <code>:latest</code>.</strong> You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.</li>

<li><strong>Keeping the <code>version:</code> key at the top of the file.</strong> Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.</li>

<li><strong>No resource limits anywhere.</strong> One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.</li>

<li><strong>Running <code>docker compose up</code> from an SSH session and walking away.</strong> Without <code>-d</code> the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.</li>

<li><strong>The Compose file lives only on the server.</strong> If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.</li>

<li><strong>Pulling from Docker Hub anonymously in a deploy script.</strong> Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.</li>

<li><strong>Automatic image updates on production.</strong> Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.</li>
</ul>



<h2 class="wp-block-heading">Best practices for Docker Compose in production</h2>



<ul class="wp-block-list">
<li><strong>One base file plus a production override.</strong> Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then <code>docker compose -f compose.yaml -f compose.prod.yaml up -d</code> is your deploy, and the diff between environments is a file you can read.</li>

<li><strong>Give every service a health check.</strong> It gates startup order, it makes <code>--wait</code> meaningful, and it gives you a status worth alerting on.</li>

<li><strong>Set memory and CPU limits on everything.</strong> Watch <code>docker stats</code> under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.</li>

<li><strong>Segment your networks.</strong> An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.</li>

<li><strong>Back up volumes, and restore one.</strong> A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.</li>

<li><strong>Make the deploy a script, not a memory.</strong> Pull, up with <code>--wait</code>, verify, prune. Six lines in the repo beats six commands somebody half-remembers.</li>

<li><strong>Run <code>docker compose config</code> in CI.</strong> It catches a malformed override before it reaches the server, and it costs nothing.</li>
</ul>



<h2 class="wp-block-heading">When to stop using Compose</h2>



<p class="wp-block-paragraph">Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is &#8220;survive the loss of this machine&#8221;, you have outgrown it, and no amount of configuration closes that gap.</p>



<p class="wp-block-paragraph">The signals I treat as a real trigger, rather than as an excuse to rewrite everything:</p>



<ul class="wp-block-list">
<li>You need more than one machine for availability, not just for capacity.</li>

<li>You need automatic rescheduling when a host dies, without somebody logging in.</li>

<li>You are scaling components independently and often enough that doing it by hand is a real cost.</li>

<li>Several teams deploy to the same infrastructure and need isolation from each other.</li>
</ul>



<p class="wp-block-paragraph">If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Docker Compose production ready?</h3>



<p class="wp-block-paragraph">For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.</p>



<h3 class="wp-block-heading">How do I get zero-downtime deployments with Docker Compose?</h3>



<p class="wp-block-paragraph">Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.</p>



<h3 class="wp-block-heading">Does <code>docker compose down</code> delete my database?</h3>



<p class="wp-block-paragraph">Not by itself. Plain <code>down</code> removes containers and the project&#8217;s networks and leaves named volumes in place. Adding <code>-v</code> removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.</p>



<h3 class="wp-block-heading">Why does my container keep restarting with exit code 137?</h3>



<p class="wp-block-paragraph">137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check <code>docker stats</code> for growth over time and <code>dmesg</code> for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.</p>



<h3 class="wp-block-heading">Should I still write <code>version:</code> at the top of my Compose file?</h3>



<p class="wp-block-paragraph">No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.</p>



<h3 class="wp-block-heading">Can I use Docker secrets without Swarm?</h3>



<p class="wp-block-paragraph">Yes, using file-based secrets. Declare a top-level <code>secrets</code> block with a <code>file:</code> source, grant it to a service, and the content appears at <code>/run/secrets/&lt;name&gt;</code> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of <code>docker inspect</code>.</p>



<h3 class="wp-block-heading">How many services is too many for one Compose file?</h3>



<p class="wp-block-paragraph">There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team&#8217;s deploy restarts another team&#8217;s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn&#8217;t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.</p>



<p class="wp-block-paragraph">Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on <code>down</code>, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Compose setup?</h2>



<p class="wp-block-paragraph">I work with teams running containerised workloads on their own servers. Things I am usually brought in for:</p>



<ul class="wp-block-list">
<li>Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first</li>

<li>Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI</li>

<li>Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does</li>

<li>Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall</li>

<li>Volume backup and restore that has been tested by restoring, not just by running</li>

<li>An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down</li>
</ul>



<p class="wp-block-paragraph">If something in your stack is behaving oddly, send me the compose file, the output of <code>docker compose ps --all</code>, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/docker-compose-in-production/">Docker Compose in Production: What Works and What Quietly Burns You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Cluster Becomes the Product: When Kubernetes Is Overkill</title>
		<link>https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 15:26:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=93</guid>

					<description><![CDATA[<p>Three hours into an ingress controller upgrade for an application that is one Go binary and a Postgres database. Nothing is broken. The cluster has just become the thing you operate. An honest look at what Kubernetes costs, when it earns that cost, and what to run instead.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/">The Cluster Becomes the Product: When Kubernetes Is Overkill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It&#8217;s a Tuesday and you&#8217;re three hours into an ingress controller upgrade. The CRD schema changed, the annotations you copied from a blog post two years ago are deprecated, and cert-manager wants a newer API version than the one your Helm chart pins.</p>



<p class="wp-block-paragraph">The application behind all of this is one Go binary and a Postgres database. It serves a few hundred requests a minute. Nobody has touched its code in a fortnight.</p>



<p class="wp-block-paragraph">That is the moment worth noticing: <strong>when Kubernetes is overkill</strong>, the tell is not that anything is broken. Everything works. It is that the cluster has quietly become the thing you operate, and the product has become a tenant.</p>



<h2 class="wp-block-heading">What it actually buys you</h2>



<p class="wp-block-paragraph">Worth being fair, because the case for Kubernetes is real and I reach for it regularly.</p>



<p class="wp-block-paragraph">It gives you bin-packing across many workloads, so a fleet of services shares machines efficiently. It gives you a declarative API that a platform team can build on, so twelve product teams deploy without twelve bespoke pipelines. It gives you self-healing, rolling updates, horizontal autoscaling and namespace-level isolation as defaults rather than as things you script. And it gives you a portable vocabulary: an engineer who knows Kubernetes knows most of your infrastructure on day one.</p>



<p class="wp-block-paragraph">Every one of those benefits scales with the number of services and the number of teams. None of them scale with how modern you want to feel.</p>



<h2 class="wp-block-heading">The bill nobody itemises</h2>



<p class="wp-block-paragraph">The cost is not the control plane fee. It is the surface area you have agreed to keep current.</p>



<p class="wp-block-paragraph">Kubernetes ships three minor releases a year and supports each for about fourteen months, and you can skip at most one minor version when upgrading. So there is a permanent, non-negotiable upgrade cadence: roughly every four months something needs planning, testing and a maintenance window, forever. Fall behind and you are doing multi-hop upgrades on an unsupported version, which is the worst combination available.</p>



<p class="wp-block-paragraph">Underneath that sit the components nobody counts when they say &#8220;we&#8217;ll just run Kubernetes&#8221;: a CNI plugin, a CSI driver, an ingress controller, cert-manager, external-dns, a metrics pipeline, an autoscaler, RBAC, and whatever operators your databases need. Each has its own release cycle, its own breaking changes and its own opinions about the others. That is where the Tuesday goes.</p>



<p class="wp-block-paragraph">Then there is debugging. A request that used to fail in one place now fails in one of several, and telling apart an application bug from a NetworkPolicy, a readiness probe, a DNS caching issue or a node under memory pressure is a genuine skill. On a good team that skill exists. On a three-person team it exists in one person&#8217;s head and goes on holiday with them.</p>



<h2 class="wp-block-heading">Signals you don&#8217;t need it yet</h2>



<ul class="wp-block-list">
<li>You have fewer than about five services, and most of them are one binary and a database.</li>
<li>One team owns everything, so there is nothing to isolate from anyone.</li>
<li>Traffic is predictable enough that you size for peak and stop thinking about it.</li>
<li>Nobody on the team has run a cluster in production before.</li>
<li>Your deploys are already fine. Nobody is complaining about them.</li>
<li>You cannot name the specific problem Kubernetes is solving, only the general one.</li>
<li>The cluster would run on two nodes, which means you have a scheduler for a decision with two possible answers.</li>
</ul>



<h2 class="wp-block-heading">Signals you probably do</h2>



<ul class="wp-block-list">
<li>Multiple teams deploying independently and tripping over each other&#8217;s environments.</li>
<li>Enough services that per-service deployment scripts have become their own maintenance problem.</li>
<li>Genuinely variable load where autoscaling saves real money, not hypothetical money.</li>
<li>A compliance or tenancy requirement that maps cleanly onto namespaces and network policy.</li>
<li>You are already running an operator-based product, like a database platform, that expects Kubernetes.</li>
<li>Somebody&#8217;s whole job is the platform, and it is not also somebody&#8217;s whole job to ship features.</li>
</ul>



<p class="wp-block-paragraph">That last one is the honest test. Kubernetes is not expensive to install. It is expensive to own, and ownership needs a name attached to it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What I&#8217;d reach for instead</h2>



<p class="wp-block-paragraph">Roughly in order of how much you take on.</p>



<p class="wp-block-paragraph"><strong>One server, systemd, and a reverse proxy.</strong> Underrated to the point of being contrarian, and completely adequate for a large number of real businesses. A VPS from Hetzner, InterServer or DigitalOcean, nginx or Caddy in front, systemd units for your services, automated backups off the box. Restart policies and health checks are built in; you are not going without them, you are just getting them from init instead of from a control loop.</p>



<p class="wp-block-paragraph"><strong>Docker Compose on one box.</strong> The step up when you want containers without a scheduler. The entire deployment is a file you can read in one screen:</p>



<pre class="wp-block-code"><code>services:
  app:
    image: ghcr.io/acme/api:1.4.2      # pinned, deployed by changing this line
    restart: always
    ports: ["127.0.0.1:8080:8080"]     # proxy handles TLS and the public port
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16                 # pin the major, upgrade deliberately
    restart: always
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes:
  pgdata:</code></pre>



<p class="wp-block-paragraph">The honest limitation: one box is one box. Losing it is an outage, and a redeploy is a brief gap unless you put something in front. For plenty of internal tools and early products, that is an acceptable trade stated out loud rather than an oversight.</p>



<p class="wp-block-paragraph"><strong>A platform that runs containers for you.</strong> Cloud Run, App Runner, Fly.io, Render, Railway. You hand over a container image and get scaling, TLS, rolling deploys and health checks without operating any of it. This is the option most teams should look at hardest, because it buys back the majority of what Kubernetes offers at almost none of the operational cost. You pay in per-unit pricing and in reduced control, and both are usually worth it below a certain scale.</p>



<p class="wp-block-paragraph"><strong>ECS on Fargate, or Nomad.</strong> Real schedulers with a fraction of the concepts. If you need orchestration but not the ecosystem, these are legitimate destinations rather than waypoints on the road to Kubernetes.</p>



<p class="wp-block-paragraph">And if you have decided Kubernetes genuinely is the answer, use a managed control plane, and consider a lightweight distribution such as k3s for small or edge deployments. Do not build the cluster yourself to save money. That is the most expensive saving in infrastructure.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><em>&#8220;We&#8217;ll need it eventually.&#8221;</em> Possibly. Adopting it three years early costs three years of upgrade cycles to buy an option you may not exercise, and the migration is not much harder later than it is now.</p>



<p class="wp-block-paragraph"><em>&#8220;It&#8217;s the industry standard.&#8221;</em> For orchestrating fleets, yes. Most applications are not fleets. Standard does not mean universally appropriate, and the same argument would put a load balancer in front of a single server.</p>



<p class="wp-block-paragraph"><em>&#8220;It avoids vendor lock-in.&#8221;</em> Partly true and frequently oversold. You have swapped a cloud provider&#8217;s API for a distribution, a set of controllers, and a pile of YAML that encodes assumptions about all of them. Portability lives in your container images and your data, not in the scheduler.</p>



<p class="wp-block-paragraph"><em>&#8220;It&#8217;ll help us hire.&#8221;</em> It helps you hire people who want to run Kubernetes. Whether that is the same as the people who will build your product is worth a moment&#8217;s thought.</p>



<h2 class="wp-block-heading">You&#8217;re not painting yourself into a corner</h2>



<p class="wp-block-paragraph">This is the part that makes the decision easy to defer, and it is the most useful thing in this post.</p>



<p class="wp-block-paragraph">The durable investments are containerising your application, externalising configuration into environment variables, keeping state out of the application and in a database or object storage, exposing a health endpoint, and logging to stdout. Do those and your app already satisfies most of what a Kubernetes manifest asks for. The scheduler underneath is genuinely swappable.</p>



<p class="wp-block-paragraph">So the choice is not Kubernetes now or a painful rewrite later. It is Kubernetes now, or the same containers running somewhere simpler until the complexity is earned.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How many services before Kubernetes makes sense?</h3>



<p class="wp-block-paragraph">There is no clean number, and anyone offering one is guessing. The better question is how many teams deploy independently. One team can coordinate without a platform. Four cannot, and that is when the abstraction starts paying for itself.</p>



<h3 class="wp-block-heading">Is Docker Compose really production-ready?</h3>



<p class="wp-block-paragraph">On a single host, for a workload that tolerates brief restarts, yes. It runs plenty of real systems. What it does not give you is failover across machines, so be clear that losing the host means an outage and decide whether that is acceptable rather than discovering it.</p>



<h3 class="wp-block-heading">Doesn&#8217;t managed Kubernetes remove the operational burden?</h3>



<p class="wp-block-paragraph">It removes the control plane, which is the part that was never your main problem. Ingress, storage, networking, upgrades and the add-on ecosystem all remain yours. Managed offerings genuinely help, and they narrow the gap rather than closing it.</p>



<h3 class="wp-block-heading">What about autoscaling? I can&#8217;t get that elsewhere.</h3>



<p class="wp-block-paragraph">You can. Cloud Run, App Runner, Fly and ECS all autoscale, several of them to zero, with no cluster to maintain. Autoscaling is a common reason to reach for Kubernetes and one of the weakest, because it is the feature most thoroughly commoditised elsewhere.</p>



<h3 class="wp-block-heading">How hard is it to migrate later?</h3>



<p class="wp-block-paragraph">Much easier than migrating off it. If your app is containerised, stateless, configured by environment and observable, moving to Kubernetes is mostly writing manifests. The hard parts of that migration are the ones you should be fixing anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Kubernetes is not complicated by accident. It is complicated because it solves a genuinely complicated problem, and if you do not have that problem you are paying the complexity and collecting none of the benefit.</p>



<p class="wp-block-paragraph">So make it a decision with a reason attached. Write down the specific thing you cannot do today, and check whether a cluster is the cheapest way to fix it. Sometimes it is. Often the honest answer is one server, a reverse proxy, and an afternoon spent on backups instead.</p>



<h2 class="wp-block-heading">Trying to decide?</h2>



<p class="wp-block-paragraph">I get called in on both sides of this: teams drowning in a cluster they did not need, and teams who genuinely need one and are putting it off. Work I take on:</p>



<ul class="wp-block-list">
<li>A straight assessment of whether Kubernetes fits your team and workload, with the reasoning written down rather than an opinion delivered.</li>
<li>Building the simpler thing properly: single-host or Compose deployments with TLS, health checks, deploys and backups that hold up in production.</li>
<li>Moving off an over-provisioned cluster onto something proportionate, without downtime.</li>
<li>Getting an application genuinely portable so the decision stays reversible: containerised, stateless, configured by environment, observable.</li>
<li>Setting up managed Kubernetes properly when it is the right answer, including the upgrade process nobody plans for.</li>
<li>Reviewing the cost, in money and in engineering hours, of what you currently run.</li>
</ul>



<p class="wp-block-paragraph">Tell me how many services you run and how many people deploy them, and I will tell you what I would build.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/">The Cluster Becomes the Product: When Kubernetes Is Overkill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Two Sources of Truth: Git-Based WordPress Deployment From Local to Production</title>
		<link>https://john-nessime.com/blog/devops/git-based-wordpress-deployment/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:28 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[Bedrock]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Composer]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[Staging]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[WordPress Database]]></category>
		<category><![CDATA[WordPress Hosting]]></category>
		<category><![CDATA[WordPress Migration]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=60</guid>

					<description><![CDATA[<p>A plugin quietly reverts after every deploy and nobody knows why. That is what happens when Git and wp-admin both think they own the file tree. Here is how to build a deployment workflow where the repo is the source of truth and the database only ever travels one way.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/git-based-wordpress-deployment/">Two Sources of Truth: Git-Based WordPress Deployment From Local to Production</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone reports that the contact form stopped working. You check the log, and nothing has touched that plugin in three weeks. You SSH into production, look at the plugin directory, and the version on disk is older than the version in your repo. Older than production had yesterday, in fact.</p>



<p class="wp-block-paragraph">Then it clicks. Somebody logged into wp-admin last month, saw an update notice, clicked it. The plugin got updated on the server and never in git. Last night&#8217;s deploy synced the repo over the top and put the old version back, along with the vulnerability the update was patching. No error, no warning. The deploy did exactly what it was told.</p>



<p class="wp-block-paragraph">That is the failure mode that kills most attempts at <strong>Git-based WordPress deployment</strong>, and it has nothing to do with your pipeline. WordPress is designed to modify itself. It installs plugins, updates core, writes files, all from the browser. The moment you put it under version control you have two systems that both believe they own the file tree, and they will quietly overwrite each other until somebody notices.</p>



<p class="wp-block-paragraph">This post covers the whole chain: what belongs in the repo, how to shut the second write path, how configuration differs per environment, which way the database is allowed to travel, and how to make staging safe enough that you can restore production data into it without emailing your customers by accident.</p>



<h2 class="wp-block-heading">The two things that make WordPress awkward under Git</h2>



<p class="wp-block-paragraph">Everything in this workflow follows from two facts.</p>



<p class="wp-block-paragraph"><strong>State is split across files and a database.</strong> Your theme is code. Your posts are data. But so are your plugin settings, your menus, your widget layouts, and your permalink structure, and those live in the database next to the content. Git only manages half your site, and it is not obvious which half a given change landed in.</p>



<p class="wp-block-paragraph"><strong>The two halves travel in opposite directions.</strong> Code moves forward, local to staging to production. Content moves backward, production to staging to local. Once a site is live, production is the only authoritative source of orders, comments and posts, and pushing a database upward destroys everything written since your last pull.</p>



<p class="wp-block-paragraph">Write that rule somewhere the whole team can see it. Almost every catastrophic WordPress deployment story is somebody pushing a database in the wrong direction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what actually goes in the repo</h2>



<p class="wp-block-paragraph">Three approaches, in increasing order of discipline and payoff. Pick one deliberately rather than drifting into the first.</p>



<ol class="wp-block-list">
<li><strong>Theme only.</strong> The repo contains one custom theme, sometimes a custom plugin. Everything else is installed through wp-admin. Easy to start, and it means you have no reproducible build: rebuilding the site from the repo is impossible.</li>
<li><strong>All of wp-content, minus uploads.</strong> Themes, plugins and mu-plugins in git; core and media excluded. This is the pragmatic middle and where most teams land. You get reproducible plugin versions and a real diff when a plugin changes.</li>
<li><strong>Composer-managed, Bedrock-style.</strong> Core, plugins and themes are all declared as dependencies. The repo holds a <code>composer.json</code> and your own code, nothing else. The lockfile is the source of truth and <code>composer install</code> rebuilds the site anywhere.</li>
</ol>



<p class="wp-block-paragraph">Option three is the one I reach for on anything with more than one developer, because &#8220;which plugin version is production running&#8221; becomes a question with an answer. The honest cost: premium plugins that are not on a Composer repository need either a private repository or a manual step, and that friction is real. If most of your plugin stack is commercial, option two is a reasonable place to stop.</p>



<p class="wp-block-paragraph">One thing to check if you are following an older tutorial. The Composer repository landscape for WordPress plugins shifted recently: WPackagist was acquired by WP Engine, and the Roots team launched WP Packages as an independent alternative with different package naming. Both work. Any guide written before that change will use the older <code>wpackagist-plugin/</code> prefix, so make sure the repository URL and the prefixes in your <code>composer.json</code> match each other.</p>



<h3 class="wp-block-heading">The gitignore that matters</h3>



<pre class="wp-block-code"><code># Secrets and machine-specific config. Never committed.
wp-config.php
.env

# Content, not code. This gets pulled down from production,
# it never gets deployed up.
wp-content/uploads/

# Generated at runtime or by the build.
wp-content/cache/
wp-content/upgrade/
wp-content/debug.log
node_modules/
vendor/</code></pre>



<p class="wp-block-paragraph">Ignoring <code>vendor/</code> assumes your deploy runs <code>composer install</code>. If your pipeline just rsyncs the repo with no build step, commit it instead, or you will ship a site with no dependencies. Decide which, then be consistent, because half-and-half is how you end up with a missing autoloader at two in the morning.</p>



<h2 class="wp-block-heading">Close the second write path</h2>



<p class="wp-block-paragraph">This is the step people skip, and it is the one that makes everything else work. If production can still install and update plugins from the browser, your repo is not the source of truth. It is a suggestion.</p>



<pre class="wp-block-code"><code>// wp-config.php on production and staging.

// Removes the ability to install, update or delete plugins and
// themes from the admin. Also disables the built-in file editor,
// so DISALLOW_FILE_EDIT is implied by this one.
define( 'DISALLOW_FILE_MODS', true );

// Stops core from updating itself behind your back, including
// the automatic minor updates that would otherwise change files
// your deploy is about to overwrite.
define( 'AUTOMATIC_UPDATER_DISABLED', true );</code></pre>



<p class="wp-block-paragraph">Be clear-eyed about the trade-off, because it is a real one. You have just taken away automatic security updates. That is only an improvement if you replace them with something: a scheduled dependency update job, a weekly review of the update list, a bot that opens a pull request when a plugin version changes. If you disable updates and then do nothing, you have made the site less safe, not more controlled.</p>



<p class="wp-block-paragraph">The update flow now runs through the repo. On your local machine:</p>



<pre class="wp-block-code"><code># Composer-managed: bump the lockfile, commit it, deploy.
composer update wp-plugin/some-plugin

# wp-content-in-git: let WP-CLI do the download locally,
# then commit the resulting file changes as a normal diff.
wp plugin update some-plugin</code></pre>



<p class="wp-block-paragraph">The second form is worth appreciating. A plugin update becomes a reviewable diff. When something breaks two days later, <code>git log</code> tells you exactly which files changed and <code>git revert</code> puts them back.</p>



<h2 class="wp-block-heading">Configuration per environment</h2>



<p class="wp-block-paragraph">Database credentials, salts and API keys differ per environment and none of them belong in git. Keep <code>wp-config.php</code> out of the repo and place it on each server, or use an environment file with a committed <code>.env.example</code> showing the required keys with no values.</p>



<p class="wp-block-paragraph">Core has a first-class way to tell environments apart, and it is underused:</p>



<pre class="wp-block-code"><code>// Per environment, in wp-config.php.
// Recognised values: 'local', 'development', 'staging', 'production'.
define( 'WP_ENVIRONMENT_TYPE', 'staging' );</code></pre>



<pre class="wp-block-code"><code>&lt;?php
// mu-plugin, committed to the repo, safe on every environment
// because it checks where it is running.

if ( wp_get_environment_type() !== 'production' ) {

    // Keep non-production sites out of search results.
    add_filter( 'pre_option_blog_public', '__return_zero' );

    // Short-circuit wp_mail() entirely. Returning a non-null
    // value stops WordPress before it hands anything to PHPMailer.
    add_filter( 'pre_wp_mail', '__return_false' );
}</code></pre>



<p class="wp-block-paragraph">That mail filter is the single most valuable line in this post. Restore a production database onto staging, forget this, and the first cron run mails real order confirmations, password resets and abandoned-cart nudges to real customers from a URL that does not work. It is silent until it is very loud.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Moving the database in the allowed direction</h2>



<p class="wp-block-paragraph">Set up WP-CLI aliases once and the rest becomes short commands instead of a runbook nobody follows.</p>



<pre class="wp-block-code"><code># wp-cli.yml in the project root, committed to the repo.
@staging:
  ssh: deploy@staging.example.com/var/www/staging.example.com
@production:
  ssh: deploy@example.com/var/www/example.com</code></pre>



<p class="wp-block-paragraph">Now pulling production down to your machine is one pipe. The trailing hyphen means &#8220;write the dump to standard output&#8221;, and the leading hyphen on the import means &#8220;read it from standard input&#8221;, so nothing touches disk:</p>



<pre class="wp-block-code"><code># Back up your local database first, then overwrite it.
wp db export local-backup.sql
wp @production db export - | wp db import -</code></pre>



<p class="wp-block-paragraph">The URLs inside that dump still point at production, and they are buried in serialized PHP arrays where a plain SQL find-and-replace corrupts the string length prefixes. This is exactly what <code>wp search-replace</code> exists for: it unserializes, replaces, and reserializes properly.</p>



<pre class="wp-block-code"><code># Always dry-run first and actually read the table list it prints.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid --dry-run

# Then for real.
wp search-replace 'https://example.com' 'https://example.test' 
  --all-tables-with-prefix --skip-columns=guid

wp cache flush</code></pre>



<p class="wp-block-paragraph">Two flags worth understanding rather than copying:</p>



<ul class="wp-block-list">
<li><strong><code>--skip-columns=guid</code></strong> because the <code>guid</code> column is a permanent identifier that feed readers and external systems use to decide whether they have already seen a post. Rewriting it makes subscribers re-receive your entire archive, and you find out from an angry email weeks later.</li>
<li><strong><code>--all-tables-with-prefix</code></strong> because plugins create their own tables using your prefix, and those tables are not registered with WordPress. Without this, the replacement misses them and you get half-migrated URLs in places nobody checks.</li>
</ul>



<p class="wp-block-paragraph">Uploads are the other half of the content problem. Do not put them in git; a media library will outgrow a repository quickly and every clone pays for it. Sync them separately with <code>rsync</code> when you need them, or skip them entirely locally and let a plugin proxy missing images from production. Offloading media to object storage sidesteps the question completely and is worth considering on any site with a large library.</p>



<h2 class="wp-block-heading">The deploy itself</h2>



<p class="wp-block-paragraph">Three broad options. <code>git pull</code> on the server is the simplest and the one I would move away from first: it puts a <code>.git</code> directory next to your document root, it has no build step, and a failed pull leaves the site half-updated. Rsync from CI is the pragmatic default. Atomic releases with a symlink swap, the way Deployer or Capistrano work, are the most correct because the site switches versions instantly and rollback is a symlink change.</p>



<p class="wp-block-paragraph">A workable rsync deploy from GitHub Actions, with staging on merge and production on tag:</p>



<pre class="wp-block-code"><code>name: Deploy
on:
  push:
    branches: [main]     # goes to staging
    tags: ['v*']         # goes to production

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4    # pin to a full SHA in production

      - name: Build
        run: |
          composer install --no-dev --optimize-autoloader
          npm ci
          npm run build

      - name: Sync files
        run: |
          echo "$DEPLOY_KEY" &gt; deploy_key
          chmod 600 deploy_key
          rsync -az --delete 
            -e "ssh -i deploy_key -o StrictHostKeyChecking=accept-new" 
            --exclude-from=.deployignore 
            ./ "$TARGET"
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
          TARGET: ${{ secrets.DEPLOY_TARGET }}</code></pre>



<p class="wp-block-paragraph"><code>--delete</code> is what makes the target match the source exactly, which is the entire point of a deploy. It is also the flag that will erase your uploads directory if <code>.deployignore</code> is wrong. Test the exclude list against staging before you ever point this at production, and use <code>--dry-run</code> the first time.</p>



<p class="wp-block-paragraph">Files landing on the server is not the same as a working deploy. Run a post-deploy step over SSH:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

cd /var/www/example.com

# Applies schema changes that a core or plugin version bump needs.
# Safe to run when there is nothing to do.
wp core update-db

# Rewrite rules live in the database, so a plugin that registers
# new routes needs this or you get 404s on the new URLs.
wp rewrite flush

# Options and transients are cached; stale ones survive a file deploy.
wp cache flush

# Prove the site answers before you call the deploy finished.
# --fail makes curl exit non-zero on a 5xx instead of shrugging.
curl --fail --silent --show-error https://example.com/ &gt; /dev/null</code></pre>



<p class="wp-block-paragraph">If you run PHP-FPM with OPcache, add a reset to that script. Otherwise PHP happily serves the previous version of your code from its compiled cache and you spend twenty minutes convinced the deploy did not run.</p>



<h2 class="wp-block-heading">Making staging safe to hold real data</h2>



<p class="wp-block-paragraph">Staging is only useful if it mirrors production, and the moment it does it becomes dangerous. Every one of these has bitten somebody:</p>



<ul class="wp-block-list">
<li><strong>Outbound email.</strong> Covered above. Kill it at the <code>wp_mail()</code> level rather than trusting a plugin setting somebody can toggle.</li>
<li><strong>Cron.</strong> A restored production database brings production&#8217;s scheduled jobs with it. Set <code>DISABLE_WP_CRON</code> on staging and do not add a system cron job unless you specifically need to test scheduling.</li>
<li><strong>Payment gateways.</strong> The database carries live API keys. Overwrite them with test-mode credentials as part of your restore script, not by hand afterwards.</li>
<li><strong>Search indexing.</strong> Filter <code>blog_public</code> as shown, and put HTTP basic auth in front of the whole environment. Duplicate content ranking above your real site is an avoidable embarrassment.</li>
<li><strong>Personal data.</strong> A production restore is real customer data on a server that probably has weaker access controls. Either anonymise user records during the restore or treat staging with the same seriousness as production.</li>
</ul>



<p class="wp-block-paragraph">Write the restore as a single script that does all of this in order, so nobody has to remember step four. If it is a checklist in a wiki, it will be skipped.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">The deploy ran but the site is unchanged</h3>



<p class="wp-block-paragraph">In order: OPcache serving compiled bytecode, a page cache or CDN holding the old response, or rsync excluding the directory you edited. Check them in that order. Confirm the file on disk actually changed before blaming anything upstream of it.</p>



<h3 class="wp-block-heading">White screen immediately after deploying</h3>



<p class="wp-block-paragraph">Usually a missing autoloader, meaning <code>vendor/</code> was neither committed nor built. Check the PHP error log rather than guessing. A memory limit lower on the server than locally is the other common cause, and it produces the same blank page.</p>



<h3 class="wp-block-heading">Site works, admin redirects in a loop</h3>



<p class="wp-block-paragraph"><code>siteurl</code> and <code>home</code> in the options table disagree with the URL you are actually using. Almost always a search-replace that missed. Check with <code>wp option get siteurl</code> and <code>wp option get home</code>, and set them explicitly with <code>wp option update</code> rather than running the replacement again.</p>



<h3 class="wp-block-heading">Serialized data broken after a migration</h3>



<p class="wp-block-paragraph">Somebody ran <code>sed</code> or a SQL <code>REPLACE</code> on the dump. The length prefix inside each serialized string no longer matches the string, so PHP silently fails to unserialize and widgets, theme options and ACF fields come back empty. Restore from the dump and redo it with <code>wp search-replace</code>. There is no partial recovery worth attempting.</p>



<h3 class="wp-block-heading">Permalinks return 404 after deploy</h3>



<p class="wp-block-paragraph">Rewrite rules are stored in the database and did not get regenerated. Run <code>wp rewrite flush</code>. If it recurs on every deploy, add it to the post-deploy script permanently.</p>



<h3 class="wp-block-heading">Plugin reverts to an older version after every deploy</h3>



<p class="wp-block-paragraph">Somebody is still updating in wp-admin. That is not a pipeline bug, it is the second write path being open. Set <code>DISALLOW_FILE_MODS</code> and give the team a documented way to request an update.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Committing <code>wp-config.php</code>, and with it your database password and salts.</li>
<li>Putting <code>wp-content/uploads</code> in the repo, then wondering why clones take ten minutes.</li>
<li>Pushing a staging database over production because &#8220;the content is the same&#8221;. It never is.</li>
<li>Using <code>sed</code> or SQL to swap URLs instead of <code>wp search-replace</code>.</li>
<li>Replacing the <code>guid</code> column, and re-sending your archive to every RSS subscriber.</li>
<li>Running <code>rsync --delete</code> against production without testing the exclude list.</li>
<li>Leaving updates enabled in wp-admin, so the repo and the server drift apart silently.</li>
<li>Disabling automatic updates without putting a review process in its place.</li>
<li>Restoring production data onto staging with mail still enabled.</li>
<li>Forgetting <code>DISABLE_WP_CRON</code> on staging, so scheduled jobs fire twice across two environments.</li>
<li>Treating &#8220;the files copied&#8221; as proof the deploy worked, with no smoke test.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Make the repo the only way code reaches a server, and enforce it with <code>DISALLOW_FILE_MODS</code>.</li>
<li>Keep code moving forward and content moving backward, and state that rule where the team can see it.</li>
<li>Deploy staging from a branch and production from a tag, so production releases are deliberate.</li>
<li>Dry-run every search-replace and every rsync the first time.</li>
<li>Back up the target database immediately before any deploy that touches it.</li>
<li>Script the staging restore end to end, including mail, cron, keys and anonymisation.</li>
<li>Finish every deploy with a real HTTP request through the public URL, not a file listing.</li>
<li>Use environment-aware code with <code>wp_get_environment_type()</code> instead of commented-out blocks.</li>
<li>Keep uploads out of git and sync or offload them separately.</li>
<li>Replace automatic updates with a scheduled dependency review, and put it in a calendar.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should WordPress core be in the repository?</h3>



<p class="wp-block-paragraph">Only if you are not managing it another way. Committing core works and makes the repo self-contained, at the cost of noisy diffs on every update. Declaring core as a Composer dependency is cleaner and gives you a pinned version without the noise. What you should not do is leave core unmanaged, so nobody can say which version production runs.</p>



<h3 class="wp-block-heading">How do I handle the database in a Git workflow?</h3>



<p class="wp-block-paragraph">You do not put it in Git. Git handles code; the database is handled by scripted pulls in one direction. Production to staging to local, with a search-replace on import. The only time a database legitimately goes upward is the initial launch, before there is real content to lose.</p>



<h3 class="wp-block-heading">What about plugin settings changed in staging?</h3>



<p class="wp-block-paragraph">This is the genuinely hard part and there is no clean answer. Settings live in the database, so they do not deploy. Options are: reapply them by hand in production with a documented checklist, script them as WP-CLI commands in a deploy step, or use a plugin that exports configuration to files. Whichever you pick, write it down, because &#8220;I&#8217;ll remember&#8221; is how a setting gets applied to staging and never production.</p>



<h3 class="wp-block-heading">Is Bedrock necessary?</h3>



<p class="wp-block-paragraph">No. It is a well-made set of defaults for Composer-managed WordPress with environment-based configuration, and it saves assembling those pieces yourself. Plenty of teams run a perfectly disciplined Git deployment with a standard directory layout. Adopt it if you want the conventions; do not adopt it as a prerequisite.</p>



<h3 class="wp-block-heading">Do I need three environments?</h3>



<p class="wp-block-paragraph">You need local and production. Staging earns its cost when you have multiple people, a client who signs off on changes, or a site where breakage costs money. On a personal blog it is overhead. On a store, skipping it means production is your test environment.</p>



<h3 class="wp-block-heading">Can I do this on shared hosting?</h3>



<p class="wp-block-paragraph">Partly. You need SSH and ideally WP-CLI, and plenty of shared plans have neither. Some managed WordPress hosts offer a git push deployment target, which handles the file half for you. If you want the full workflow with your own build steps and post-deploy hooks, a small VPS from a provider like InterServer gives you the SSH access and cron control the workflow assumes.</p>



<h3 class="wp-block-heading">How do I roll back a bad deploy?</h3>



<p class="wp-block-paragraph">Files are easy: redeploy the previous tag, or swap the symlink back if you use atomic releases. The database is the hard part, which is why you take a dump immediately before any deploy that runs a migration. Rolling files back without rolling the database back can leave you worse off than the broken version, so know which of the two actually changed before you act.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Git-based WordPress deployment is not really a pipeline problem. The pipeline is the easy part, and you can build a working one in an afternoon. The hard part is making the repo the only way code reaches a server, and then keeping the database flowing in exactly one direction.</p>



<p class="wp-block-paragraph">Get those two right and everything else is detail. Get them wrong and you will keep having the same confusing morning, staring at a plugin version that nobody remembers changing.</p>



<h2 class="wp-block-heading">Want this set up properly on your site?</h2>



<p class="wp-block-paragraph">Retrofitting version control onto a live WordPress site is fiddly work, mostly because you have to do it without downtime and without losing whatever has been changed directly on the server. Things I take on:</p>



<ul class="wp-block-list">
<li>Moving an existing production site into Git without downtime, including reconciling whatever has drifted on the server.</li>
<li>Converting a site to Composer-managed plugins and core, with a working lockfile and a sane update process.</li>
<li>Building local, staging and production environments that actually match, with per-environment configuration.</li>
<li>Deploy pipelines in GitHub Actions or GitLab CI: build, rsync or atomic releases, post-deploy WP-CLI steps, smoke test, rollback path.</li>
<li>Scripted database refreshes from production to staging with mail, cron, API keys and personal data handled automatically.</li>
<li>Server setup for the workflow: SSH deploy users, correct file ownership, OPcache resets, cron.</li>
</ul>



<p class="wp-block-paragraph">Tell me how your site is hosted and how plugins currently get updated, and I will tell you what the first step should be.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/git-based-wordpress-deployment/">Two Sources of Truth: Git-Based WordPress Deployment From Local to Production</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</title>
		<link>https://john-nessime.com/blog/devops/github-actions-vps-deployment/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:21 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[rsync]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=66</guid>

					<description><![CDATA[<p>A deploy key in GitHub secrets is a shell on your production server, handed to a container you don't control, running code from maintainers you've never met. Here's how to build the pipeline so a leaked key isn't worth much: forced commands, scoped sudo, pinned actions and a real approval gate.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Stop and look at what you are about to build. A private SSH key that can log into your production server, pasted into a text box on github.com, handed to a container you do not control, which then runs code from repositories maintained by people you have never met.</p>



<p class="wp-block-paragraph">That is not hypothetical caution. In March 2025 a widely used third-party action was compromised: the attacker got hold of a maintainer&#8217;s token, pushed a malicious commit, and then <em>retroactively repointed every version tag</em> at it. Repositories that referenced the action by tag, which is nearly all of them, picked up the payload automatically. It dumped the runner&#8217;s secrets into the build log. On public repositories, those logs are readable by anyone.</p>



<p class="wp-block-paragraph">Nobody&#8217;s workflow file changed. Nobody clicked anything. The pipeline just quietly handed out its credentials one Friday.</p>



<p class="wp-block-paragraph">A <strong>GitHub Actions VPS deployment</strong> is fundamentally a credential handoff, and most guides on the topic skip straight to the YAML. This post covers the YAML too, but the useful part is what surrounds it: how to narrow the key so it cannot do much, how to verify you are talking to the right server, how to stop an unreviewed commit reaching production, and which of the failures here are silent.</p>



<h2 class="wp-block-heading">Who can actually reach that key</h2>



<p class="wp-block-paragraph">Worth being explicit, because it is a longer list than people expect:</p>



<ul class="wp-block-list">
<li><strong>Anyone with write access to the repository.</strong> A workflow file is code. Push a branch with a step that prints the key somewhere, trigger it, done. Write access to the repo is effectively access to the server.</li>
<li><strong>Every action in the job.</strong> Secrets are exposed to the whole job, not just the step that references them. A compromised action anywhere in that job can read the environment.</li>
<li><strong>Anyone who can influence what runs.</strong> This is why <code>pull_request_target</code> combined with checking out the pull request&#8217;s own code is such a well-known foot-gun: it runs the fork&#8217;s code with the base repository&#8217;s secrets in scope.</li>
</ul>



<p class="wp-block-paragraph">The goal is not to make the key unreachable. It is to make a stolen key not worth much.</p>



<h2 class="wp-block-heading">Decide how the runner authenticates</h2>



<p class="wp-block-paragraph">Three options, and the right one depends on what you already run.</p>



<ol class="wp-block-list">
<li><strong>A dedicated SSH key in repository secrets.</strong> The default. Simple, works everywhere, and it is a long-lived credential sitting in a place several people can reach. Acceptable when the key is narrowed properly, which is most of the rest of this post.</li>
<li><strong>An overlay network.</strong> Put the runner and the server on a private network with Tailscale or WireGuard for the duration of the job, using an ephemeral auth key, and close SSH to the public internet entirely. More moving parts, and it removes port 22 from the internet rather than defending it. If you already run an overlay for other reasons, this is clearly better.</li>
<li><strong>A self-hosted runner on the server.</strong> No inbound SSH at all, since the runner polls outward. The trade-off is real and often understated: you have just installed a general-purpose code execution service on your production box, and for a public repository that is close to indefensible. On a private repository with a trusted team it is reasonable.</li>
</ol>



<p class="wp-block-paragraph">One thing that sounds appealing and is not practical: allowlisting GitHub&#8217;s runner IP ranges at your firewall. The ranges are published through GitHub&#8217;s meta API, but they are enormous and they change, and every other GitHub customer&#8217;s job runs from inside them. It buys you very little for a lot of maintenance.</p>



<h2 class="wp-block-heading">Narrow the key until it is boring</h2>



<p class="wp-block-paragraph">Assume the key leaks. What can the holder do? Work backwards from there.</p>



<p class="wp-block-paragraph"><strong>Give it its own user.</strong> A <code>deploy</code> account that owns the application directory and nothing else. Not your admin account, not root, and definitely not a key you also use from your laptop.</p>



<p class="wp-block-paragraph"><strong>Restrict the key in <code>authorized_keys</code>.</strong> SSH lets you attach options to an individual key, and they apply regardless of what the client asks for:</p>



<pre class="wp-block-code"><code># ~deploy/.ssh/authorized_keys
#
# restrict = no port forwarding, no agent forwarding, no X11,
#            no PTY, no user rc file. Deny everything, then
#            re-enable only what you need.
# command  = ignore whatever the client asked to run and run
#            this instead. The client's command is available
#            to the script as $SSH_ORIGINAL_COMMAND.

restrict,command="/usr/local/bin/deploy-guard" ssh-ed25519 AAAA...  deploy@github-actions</code></pre>



<p class="wp-block-paragraph">The guard script decides what the key is allowed to do. Keep it short enough to read in one go:</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
# /usr/local/bin/deploy-guard
set -euo pipefail

case "${SSH_ORIGINAL_COMMAND:-}" in
  # rsync sends its own server-side invocation. Match it loosely
  # enough to work, tightly enough to pin the destination.
  "rsync --server "*" /var/www/app/")
    exec $SSH_ORIGINAL_COMMAND
    ;;
  "deploy:reload")
    exec sudo /bin/systemctl reload nginx
    ;;
  *)
    echo "refused: $SSH_ORIGINAL_COMMAND" &gt;&amp;2
    exit 1
    ;;
esac</code></pre>



<p class="wp-block-paragraph">Now a stolen key can write to one directory and reload one service. It cannot open a shell, cannot forward a port into your private network, and cannot read anything else on the box.</p>



<p class="wp-block-paragraph"><strong>Scope the sudo entry too.</strong> The deploy user needs to reload a service, not run arbitrary commands as root:</p>



<pre class="wp-block-code"><code># visudo -f /etc/sudoers.d/deploy
# One command, no password, no wildcards. A wildcard here
# often turns back into a general-purpose root shell.
deploy ALL=(root) NOPASSWD: /bin/systemctl reload nginx</code></pre>



<p class="wp-block-paragraph">Be honest about the cost: the guard script is one more thing to maintain, and the day you add a new deploy step it will refuse it and you will spend ten minutes confused. That is the trade you are making, and on anything handling customer data it is worth it. On a hobby project, a plain unrestricted <code>deploy</code> user with no sudo is a reasonable stopping point.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Verify the server, not just the key</h2>



<p class="wp-block-paragraph">Almost every tutorial on this topic contains <code>StrictHostKeyChecking=no</code>, which turns off the check that stops you handing your credentials to whatever machine answered. It is there because the alternative is mildly annoying, and the alternative takes one command.</p>



<pre class="wp-block-code"><code># Run this once, from a machine you trust, and compare the
# fingerprint against what the server reports locally.
ssh-keyscan -t ed25519 example.com</code></pre>



<p class="wp-block-paragraph">Store that output as a repository variable, not a secret. It is public information, and keeping it visible means you can actually see it when it changes, which matters because it will change the first time you rebuild the server.</p>



<h2 class="wp-block-heading">The workflow</h2>



<pre class="wp-block-code"><code>name: Deploy

on:
  push:
    tags: ['v*']          # production ships from tags, not branches

# This job reads code and talks to a server. It has no business
# writing to the repository, so take that away.
permissions:
  contents: read

concurrency:
  group: deploy-production
  # Deliberately NOT cancel-in-progress. Killing a deploy halfway
  # leaves the server in a state nobody designed.
  cancel-in-progress: false

jobs:
  deploy:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    environment:
      name: production      # scoped secrets and required reviewers live here
      url: https://example.com

    steps:
      # Pin every action to a full 40-character commit SHA, with the
      # version in a trailing comment so Dependabot can still bump it.
      # Get the SHA from the action's releases page.
      - uses: actions/checkout@PUT_THE_FULL_COMMIT_SHA_HERE  # pinned

      - name: Build
        run: |
          npm ci
          npm run build

      - name: Configure SSH
        # Pass secrets through env, never interpolate ${{ }} directly
        # into a run block. Interpolation happens before the shell
        # sees the script, so a value with the wrong characters
        # either breaks the script or becomes part of it.
        env:
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          KNOWN_HOSTS: ${{ vars.DEPLOY_KNOWN_HOSTS }}
        run: |
          install -m 700 -d ~/.ssh
          printf '%sn' "$DEPLOY_KEY" &gt; ~/.ssh/id_ed25519
          chmod 600 ~/.ssh/id_ed25519
          printf '%sn' "$KNOWN_HOSTS" &gt; ~/.ssh/known_hosts

      - name: Ship
        run: |
          rsync -az --delete --exclude-from=.deployignore 
            -e "ssh -o BatchMode=yes" 
            ./dist/ deploy@example.com:/var/www/app/

      - name: Reload and verify
        run: |
          ssh -o BatchMode=yes deploy@example.com 'deploy:reload'
          curl --fail --silent --show-error https://example.com/healthz &gt; /dev/null</code></pre>



<p class="wp-block-paragraph">Three details in there worth calling out, because they are the ones people leave out.</p>



<p class="wp-block-paragraph"><code>-o BatchMode=yes</code> stops SSH prompting for anything. Without it, a missing key or an unknown host produces a prompt that nobody is there to answer, and the job hangs until the timeout instead of failing in three seconds.</p>



<p class="wp-block-paragraph"><code>--delete</code> is what makes the target match the source, which is the point of a deploy, and also what will erase your uploads directory if <code>.deployignore</code> is wrong. Run it once with <code>--dry-run</code> against staging and read the list before you point it at production.</p>



<p class="wp-block-paragraph">The final <code>curl --fail</code> is the difference between &#8220;the files copied&#8221; and &#8220;the site works&#8221;. Files landing on disk is not a successful deploy, and a job that goes green while the site returns 502 is worse than one that fails.</p>



<h2 class="wp-block-heading">Third-party actions</h2>



<p class="wp-block-paragraph">There are convenient actions that wrap SSH deployment, and they save maybe eight lines. In the one job on your account that holds a production credential, eight lines is a bad trade for another maintainer in your trust chain. The workflow above uses plain <code>ssh</code> and <code>rsync</code> for that reason.</p>



<p class="wp-block-paragraph">Where you do use third-party actions, pin them to a full commit SHA. A tag is a pointer and pointers can be moved, which is precisely what happened in the incident at the top of this post. A SHA is content-addressed: it cannot be repointed at different code.</p>



<p class="wp-block-paragraph">Two things make this sustainable rather than miserable. Dependabot understands SHA pins with a version comment and will open pull requests to bump them. And you can restrict which actions are allowed to run at all, in the repository or organisation settings, which is worth doing once and forgetting about.</p>



<h2 class="wp-block-heading">Gate production properly</h2>



<p class="wp-block-paragraph">The <code>environment:</code> key in that workflow is doing more work than it appears to. Environments give you three things you cannot get from repository secrets alone:</p>



<ul class="wp-block-list">
<li><strong>Scoped secrets.</strong> The production key is only available to jobs that declare the production environment. A branch that adds a job without it gets nothing.</li>
<li><strong>Required reviewers.</strong> The job pauses and waits for a named human to approve it. This is the single control that turns &#8220;anyone with write access can deploy&#8221; into &#8220;anyone with write access can request a deploy&#8221;.</li>
<li><strong>Deployment branch and tag rules.</strong> Restrict the environment so only protected tags or a specific branch can use it, so a feature branch cannot reach production even with the right job definition.</li>
</ul>



<p class="wp-block-paragraph">Deploying from tags rather than branch pushes is worth the small friction. It makes a release a deliberate act, and it gives you an obvious rollback: redeploy the previous tag.</p>



<p class="wp-block-paragraph">For anything with real traffic, put the files somewhere new and switch atomically rather than rsyncing over a live directory. Copy into a timestamped release directory, then move a symlink. The site changes version instantly instead of spending several seconds as a mix of old and new files, and rollback becomes a symlink change rather than a redeploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Permission denied (publickey)</h3>



<p class="wp-block-paragraph">The most common cause by far is a missing trailing newline. Private keys must end with one, and pasting into a web form frequently drops it. That is why the workflow uses <code>printf '%sn'</code> rather than <code>echo</code>: it guarantees the newline regardless of what got stored.</p>



<p class="wp-block-paragraph">After that, check permissions on the server: <code>~/.ssh</code> must be 700 and <code>authorized_keys</code> 600, owned by the deploy user, and the home directory must not be group-writable. Add <code>-vvv</code> to the ssh command to see how far the handshake gets, and read <code>journalctl -u ssh</code> on the server for the reason, since the client is deliberately vague.</p>



<h3 class="wp-block-heading">Host key verification failed</h3>



<p class="wp-block-paragraph">Either the <code>known_hosts</code> variable is empty, or the server&#8217;s host key genuinely changed. If you rebuilt the server, regenerate it with <code>ssh-keyscan</code> and update the variable. If you did not rebuild the server, stop and find out why the key changed before you update anything.</p>



<h3 class="wp-block-heading">The job hangs until the timeout</h3>



<p class="wp-block-paragraph">SSH is waiting on a prompt nobody will answer. Add <code>-o BatchMode=yes</code> and it fails immediately with a readable reason instead.</p>



<h3 class="wp-block-heading">rsync works from my laptop, fails from the runner</h3>



<p class="wp-block-paragraph">If you set a forced command, it is rejecting the invocation. The refusal message goes to stderr on the server side, so add a log line to the guard script that records <code>$SSH_ORIGINAL_COMMAND</code>, run the job once, and read what rsync actually sent.</p>



<h3 class="wp-block-heading">Works on push, fails on pull requests</h3>



<p class="wp-block-paragraph">Working as intended. Secrets are not exposed to workflows triggered by pull requests from forks. Do not reach for <code>pull_request_target</code> to work around it. Split the workflow so that pull requests build and test without secrets, and only pushes or tags deploy.</p>



<h3 class="wp-block-heading">A secret appeared in the logs</h3>



<p class="wp-block-paragraph">Masking matches the exact stored string. Transform it in any way and the mask stops working: base64-encoded, JSON-escaped, split across lines, or with whitespace trimmed. Do not treat masking as a control. Rotate the key, delete the run logs, and remove whatever produced the output.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using an existing personal SSH key instead of generating a dedicated one for the runner.</li>
<li>Giving the deploy user full sudo, or a sudo rule with a wildcard in it.</li>
<li>Disabling host key checking with <code>StrictHostKeyChecking=no</code>.</li>
<li>Referencing third-party actions by tag in the job that holds your production key.</li>
<li>Interpolating <code>${{ secrets.X }}</code> directly into a <code>run:</code> script instead of passing it through <code>env:</code>.</li>
<li>Leaving default write permissions on a job that only needs to read.</li>
<li>Using <code>cancel-in-progress: true</code> on a deploy, so a second push interrupts the first mid-copy.</li>
<li>Running <code>rsync --delete</code> at production without testing the exclude list.</li>
<li>Treating &#8220;the files copied&#8221; as a successful deploy, with no health check.</li>
<li>Deploying straight from branch pushes with no approval step.</li>
<li>Relying on log masking to keep a leaked secret out of the output.</li>
<li>Never rotating the deploy key, including after someone leaves the team.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Dedicated deploy user, dedicated key, used for nothing else.</li>
<li>Restrict the key in <code>authorized_keys</code> with <code>restrict</code> and a forced command.</li>
<li>Scope sudo to the exact commands the deploy needs.</li>
<li>Pin every action to a full commit SHA and let Dependabot bump them.</li>
<li>Keep the deploy job small: fewer steps means fewer things that can read the secret.</li>
<li>Set <code>permissions:</code> explicitly, defaulting to read-only.</li>
<li>Verify host keys from a stored <code>known_hosts</code> value.</li>
<li>Use environments with required reviewers and deployment rules for production.</li>
<li>Deploy from tags, and make rollback a redeploy of the previous tag or a symlink swap.</li>
<li>End every deploy with a real HTTP request against the public URL.</li>
<li>Rotate the deploy key on a schedule and whenever someone leaves.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is it safe to store an SSH private key in GitHub secrets?</h3>



<p class="wp-block-paragraph">Safe enough, if the key is worth little on its own. Secrets are encrypted at rest and masked in logs, but they are readable by any job that references them and by anyone who can modify a workflow. Assume it can leak and make sure a leaked key only grants write access to one directory and one service reload.</p>



<h3 class="wp-block-heading">Should I use a ready-made SSH deploy action?</h3>



<p class="wp-block-paragraph">They work and they are convenient. My preference is plain <code>ssh</code> and <code>rsync</code> in the job that touches production, because it removes a maintainer from the trust chain for very little added effort. If you do use one, pin it to a SHA like anything else.</p>



<h3 class="wp-block-heading">Can I use OIDC instead of a stored key?</h3>



<p class="wp-block-paragraph">Not directly against sshd, which has no concept of an OIDC token. OIDC removes long-lived secrets when the target is a cloud provider that can verify GitHub&#8217;s token. To get the same benefit for a plain VPS you need something in between that trades the token for short-lived access: a secrets manager, a bastion that supports it, or an overlay network with ephemeral auth keys.</p>



<h3 class="wp-block-heading">Should I allowlist GitHub&#8217;s IP ranges on my firewall?</h3>



<p class="wp-block-paragraph">Generally not worth it. The published ranges are large and change over time, so you are signing up for ongoing maintenance, and every other GitHub customer&#8217;s runners share those ranges. Narrowing what the key can do achieves more.</p>



<h3 class="wp-block-heading">Self-hosted runner or hosted runner?</h3>



<p class="wp-block-paragraph">Hosted for public repositories, without exception, because a self-hosted runner will execute code from pull requests. Self-hosted is reasonable on a private repository with a trusted team, and it removes inbound SSH entirely. Just be clear that you have moved the risk rather than removed it.</p>



<h3 class="wp-block-heading">How do I roll back a bad deploy?</h3>



<p class="wp-block-paragraph">Redeploy the previous tag, or swap the symlink back if you use atomic releases. The part that does not roll back is the database, so if a release ran a migration, know what it did before you revert the files. Files going backwards while the schema stays forwards is often worse than the bug you were fixing.</p>



<h3 class="wp-block-heading">How often should the deploy key be rotated?</h3>



<p class="wp-block-paragraph">On a schedule you will actually keep, and immediately whenever someone with repository access leaves or a run log looks suspicious. Rotation is two steps, adding the new public key to <code>authorized_keys</code> and updating the secret, so there is no good reason for it to be a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">The YAML is the easy half. A working GitHub Actions VPS deployment takes an afternoon; the part that decides whether it is a good idea is what that key can do once it is out of your hands, and it will eventually be out of your hands.</p>



<p class="wp-block-paragraph">So build it assuming the key leaks. Dedicated user, forced command, scoped sudo, pinned actions, an approval gate on production. Then the worst case is an inconvenient rotation rather than an incident, and you will not be reading a build log at midnight trying to work out what a stranger&#8217;s code did with your credentials.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most deploy pipelines I get asked to look at work fine and hand out far more access than they need. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a GitHub Actions deploy to a VPS end to end: build, ship, reload, health check, rollback path.</li>
<li>Auditing an existing pipeline for what a leaked runner secret would actually grant, and closing the gap.</li>
<li>Locking down the server side: deploy user, forced commands, scoped sudo, atomic release directories.</li>
<li>Moving SSH off the public internet onto an overlay network, or setting up a self-hosted runner safely.</li>
<li>Pinning and auditing third-party actions, plus repository settings that limit what can run.</li>
<li>Environments, required reviewers and tag-based releases for teams that currently deploy on every push to main.</li>
</ul>



<p class="wp-block-paragraph">Send me your workflow file and the output of <code>ssh deploy@yourhost 'id'</code>, and I will tell you what a stolen key would get.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/github-actions-vps-deployment/">Your Deploy Key Is a Root Shell: GitHub Actions VPS Deployment Over SSH, Done Carefully</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
