<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Platform Engineering | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/platform-engineering/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/platform-engineering/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 27 Oct 2026 06:05:56 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Platform Engineering | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/platform-engineering/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</title>
		<link>https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 27 Oct 2026 06:05:56 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon DataZone]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudFormation]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[IAM Identity Center]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[SageMaker AI]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=513</guid>

					<description><![CDATA[<p>SageMaker Unified Studio is a front door, not a runtime. It consolidates the interface while leaving your bill, your IAM design and your infrastructure as code exactly where they were, and it quietly hands you a DataZone governance system to operate. Here is what it genuinely replaces, what it doesn't, the failure modes that bite during adoption, and how to decide whether it is worth it for your team.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question always arrives in the same shape. A team stands up a domain, everyone logs in, the demo goes well, and then somebody asks when they can turn off the Glue console and the Athena query editor.</p>



<p class="wp-block-paragraph">They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.</p>



<p class="wp-block-paragraph">This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.</p>



<h2 class="wp-block-heading">SageMaker Unified Studio is a front door, not a runtime</h2>



<p class="wp-block-paragraph">Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.</p>



<p class="wp-block-paragraph">AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project&#8217;s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new <em>authorisation</em> layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.</p>



<h2 class="wp-block-heading">What it actually replaces</h2>



<h3 class="wp-block-heading">The pile of standalone studios and query editors</h3>



<p class="wp-block-paragraph">This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.</p>



<p class="wp-block-paragraph">The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a &#8220;workspace&#8221; meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.</p>



<h3 class="wp-block-heading">Per-team domain sprawl</h3>



<p class="wp-block-paragraph">The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.</p>



<h3 class="wp-block-heading">The access request spreadsheet</h3>



<p class="wp-block-paragraph">There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.</p>



<p class="wp-block-paragraph">The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.</p>



<h2 class="wp-block-heading">What it doesn&#8217;t replace</h2>



<h3 class="wp-block-heading">Your bill, or your ability to explain it</h3>



<p class="wp-block-paragraph">There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.</p>



<p class="wp-block-paragraph">What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer&#8217;s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.</p>



<p class="wp-block-paragraph">Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.</p>



<h3 class="wp-block-heading">Your IAM and Lake Formation design</h3>



<p class="wp-block-paragraph">Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.</p>



<p class="wp-block-paragraph">Nobody in the adoption meeting will say &#8220;this fixes our permissions mess.&#8221; Plenty of people will quietly assume it. Say it out loud early.</p>



<h3 class="wp-block-heading">Your infrastructure as code</h3>



<p class="wp-block-paragraph">This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.</p>



<p class="wp-block-paragraph">It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.</p>



<h3 class="wp-block-heading">SageMaker AI</h3>



<p class="wp-block-paragraph">The original SageMaker was renamed SageMaker AI and remains available on its own. If your team&#8217;s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.</p>



<h2 class="wp-block-heading">What SageMaker Unified Studio quietly adds</h2>



<p class="wp-block-paragraph">Here is the section that rarely makes it into the migration plan.</p>



<h3 class="wp-block-heading">You now operate DataZone, intentionally or not</h3>



<p class="wp-block-paragraph">SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a <code>dzd</code> prefix, and the console for creating one still lives under the DataZone console path.</p>



<p class="wp-block-paragraph">You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:</p>



<pre class="wp-block-code"><code># Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID</code></pre>



<p class="wp-block-paragraph">Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.</p>



<h3 class="wp-block-heading">Project creation is a CloudFormation deploy wearing a friendly hat</h3>



<p class="wp-block-paragraph">Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.</p>



<p class="wp-block-paragraph">This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation &#8220;just doesn&#8217;t work,&#8221; do not reproduce it in the UI. Read the stack events:</p>



<pre class="wp-block-code"><code># Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The first failed resource causes the rollback. Everything after it is noise. This one habit turns most &#8220;the platform is broken&#8221; tickets into five-minute answers.</p>



<h3 class="wp-block-heading">A domain-type fork you will live with for a long time</h3>



<p class="wp-block-paragraph">Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.</p>



<p class="wp-block-paragraph">Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post&#8217;s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.</p>



<h2 class="wp-block-heading">Where adoption actually goes wrong</h2>



<h3 class="wp-block-heading">&#8220;It works in my console but not in the project&#8221;</h3>



<p class="wp-block-paragraph">This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.</p>



<p class="wp-block-paragraph">Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:</p>



<pre class="wp-block-code"><code>aws sts get-caller-identity</code></pre>



<p class="wp-block-paragraph">If the ARN is not the one you assumed, stop. You have found the problem.</p>



<h3 class="wp-block-heading">The bill went up after a migration that was supposed to save money</h3>



<p class="wp-block-paragraph">Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.</p>



<h3 class="wp-block-heading">Nobody can find the data that is definitely there</h3>



<p class="wp-block-paragraph">Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a migration rather than an addition.</strong> Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.</li>
<li><strong>Letting users self-serve projects on day one.</strong> Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.</li>
<li><strong>Choosing the domain authentication type in a hurry.</strong> It is the hardest thing here to change later.</li>
<li><strong>Skipping the catalog.</strong> Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.</li>
<li><strong>Assuming the CI/CD CLI provisions your platform.</strong> It deploys applications into projects that already exist.</li>
<li><strong>Not tagging project resources from the start.</strong> Retrofitting attribution across dozens of blueprint-created resources is miserable work.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">A rough order of operations for anyone weighing this up:</p>



<ol class="wp-block-list">
<li><strong>Ask what problem you are solving.</strong> If it is &#8220;our teams cannot discover or safely share each other&#8217;s data,&#8221; this is the right tool. If it is &#8220;we have too many consoles,&#8221; it is an expensive way to buy fewer bookmarks.</li>
<li><strong>Count your teams.</strong> The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.</li>
<li><strong>Decide the authentication model first.</strong> Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.</li>
<li><strong>Write the project profiles yourself.</strong> The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.</li>
<li><strong>Prove the teardown path before the build path.</strong> Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.</li>
<li><strong>Keep the old consoles.</strong> They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.</li>
</ol>



<p class="wp-block-paragraph">One more piece of judgement: wire your existing observability in from the start rather than relying on the studio&#8217;s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Unified Studio replace SageMaker Studio?</h3>



<p class="wp-block-paragraph">It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.</p>



<h3 class="wp-block-heading">Do I still need Amazon DataZone separately?</h3>



<p class="wp-block-paragraph">Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.</p>



<h3 class="wp-block-heading">How much does SageMaker Unified Studio cost?</h3>



<p class="wp-block-paragraph">The interface has no standalone charge. You pay for each AWS service you use through it on that service&#8217;s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.</p>



<h3 class="wp-block-heading">Can I manage domains and projects with Terraform?</h3>



<p class="wp-block-paragraph">AWS&#8217;s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.</p>



<h3 class="wp-block-heading">What happens to my Glue jobs and Athena queries?</h3>



<p class="wp-block-paragraph">Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.</p>



<h3 class="wp-block-heading">Is it worth adopting for a small team?</h3>



<p class="wp-block-paragraph">Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.</p>



<h3 class="wp-block-heading">Can I run both the old and new experiences at once?</h3>



<p class="wp-block-paragraph">Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.</p>



<p class="wp-block-paragraph">That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was &#8220;fewer tabs.&#8221; The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.</p>



<p class="wp-block-paragraph">Ask which one you are before the first project gets created, not after the fortieth.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your adoption plan?</h2>



<p class="wp-block-paragraph">This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup</li>
<li>Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults</li>
<li>Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use</li>
<li>Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover</li>
<li>Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants</li>
<li>Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long</li>
</ul>



<p class="wp-block-paragraph">Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</title>
		<link>https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 01 Oct 2026 06:02:12 +0000</pubDate>
				<category><![CDATA[Continuous Delivery]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Actions Runner Controller]]></category>
		<category><![CDATA[Build Cache]]></category>
		<category><![CDATA[BuildKit]]></category>
		<category><![CDATA[Cache Mounts]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[Flaky Tests]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[GitLab CI]]></category>
		<category><![CDATA[Multi-Stage Builds]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Queue Time]]></category>
		<category><![CDATA[Required Status Checks]]></category>
		<category><![CDATA[Test Parallelization]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=706</guid>

					<description><![CDATA[<p>The build duration graph looks fine and shipping is still slow. Most CI/CD pipeline bottlenecks are waiting, not working: queue time, caches that report a hit and deliver nothing, stage ordering nobody revisited, and required checks stuck at pending. Here is how to find yours.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/">CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline takes eleven minutes. The pull request took two days to merge.</p>



<p class="wp-block-paragraph">That gap is where the real cost lives, and it is almost never on anyone&#8217;s dashboard. The build duration graph looks fine. The team still says shipping is slow. Somebody proposes buying bigger runners, and six weeks later the graph has moved a little and the complaints are identical.</p>



<p class="wp-block-paragraph">Most CI/CD pipeline bottlenecks are not the thing people go looking for. Compute is usually the smallest slice. The expensive parts are minutes spent waiting for a runner, a cache that reports a hit and delivers nothing, a stage blocking four jobs that do not depend on it, and a required check sitting at &#8220;Pending&#8221; until someone notices. This post is organized by those failure families rather than by tool. Examples use GitHub Actions and GitLab CI, but the shapes apply to Jenkins, Buildkite, CircleCI and Argo Workflows too.</p>



<h2 class="wp-block-heading">First, split the number you are staring at</h2>



<p class="wp-block-paragraph">&#8220;The build takes eleven minutes&#8221; is not a measurement. It is three measurements added together and then thrown away:</p>



<ul class="wp-block-list">
<li><strong>Queue time.</strong> From job creation to a runner picking it up. You pay this in wall clock, not compute.</li>

<li><strong>Compute time.</strong> The job actually running. The only part most dashboards show.</li>

<li><strong>Gate time.</strong> Waiting on a human, an approval, a shared environment, or a check that never fired.</li>
</ul>



<p class="wp-block-paragraph">Until you split those, every optimization is a guess. On GitHub Actions the jobs API hands you the split directly, because each job carries a creation timestamp and a start timestamp:</p>



<pre class="wp-block-code"><code>gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs 
  --jq '.jobs[] | {name, created_at, started_at, completed_at}'</code></pre>



<p class="wp-block-paragraph">Creation to start is queue. Start to completion is compute. Run that across your last fifty runs on the default branch and the answer is usually sitting there in plain sight. GitLab reports the same idea on its jobs API, with a queued duration alongside the run duration. If the two numbers are roughly equal, you have a capacity problem rather than a build problem, and faster CPUs will fix half of nothing.</p>



<h2 class="wp-block-heading">Queue time: the minutes you pay for twice</h2>



<p class="wp-block-paragraph">Queue time has two causes and they need different fixes.</p>



<h3 class="wp-block-heading">Not enough runners</h3>



<p class="wp-block-paragraph">On hosted runners you are bounded by the concurrency your plan allows. On self-hosted you are bounded by whatever fleet you provisioned, which for most small teams is a fixed number of always-on boxes that sit idle overnight and saturate at 4pm.</p>



<p class="wp-block-paragraph">If you run your own fleet on Kubernetes, Actions Runner Controller is where most people land. It creates runner scale sets that scale on workflow demand and can drop to zero when the queue is empty, with ephemeral pods so every job starts clean. That last part matters more than the scaling: a persistent runner accumulates state, and state is where &#8220;works on my machine&#8221; comes back as a CI flake.</p>



<p class="wp-block-paragraph">If Kubernetes is more machinery than you want, a couple of dedicated boxes from a provider like Contabo or InterServer running the runner agent under systemd gets you a long way, with hosted runners as overflow when the queue backs up. Managed runner services such as Depot and BuildJet sell faster hardware behind the same interface. Their honest pitch is that you stop maintaining a fleet. The honest counterpoint is another vendor in the path of every merge, and a bill that scales with commit volume.</p>



<h3 class="wp-block-heading">Runners busy on work that is already obsolete</h3>



<p class="wp-block-paragraph">Check this one first because it is free. Push three commits to a branch in five minutes and by default you get three pipelines. The first two are answering a question nobody is asking anymore, and they are holding the runners the third one needs.</p>



<p class="wp-block-paragraph">On GitHub Actions, a concurrency group keyed on the ref does it:</p>



<pre class="wp-block-code"><code>concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true</code></pre>



<p class="wp-block-paragraph">Put that at workflow level on PR builds. Do not put it on a deploy workflow with cancellation enabled, or you will eventually kill a release halfway through and spend an afternoon working out what actually shipped.</p>



<p class="wp-block-paragraph">GitLab splits the same job across two settings that have to agree. The <code>interruptible</code> keyword marks a job as safe to kill and defaults to false. The workflow-level auto-cancel setting decides what happens on a new commit:</p>



<pre class="wp-block-code"><code>workflow:
  auto_cancel:
    on_new_commit: interruptible

test:
  interruptible: true
  script:
    - make test

deploy:
  interruptible: false
  script:
    - make deploy</code></pre>



<p class="wp-block-paragraph">The default mode is conservative, which means once any non-interruptible job has started, the whole pipeline stops being cancelable. Setting it to <code>interruptible</code> cancels only the jobs that opted in and leaves the rest running. That distinction is the difference between a policy that works and one that quietly does nothing, and it is the usual reason teams turn auto-cancel on and see no change.</p>



<h2 class="wp-block-heading">Caches that look like they are working</h2>



<p class="wp-block-paragraph">My favorite family of CI/CD pipeline bottlenecks, because the failure is completely silent. The step is green. The log says the cache was restored. The install takes exactly as long as before.</p>



<h3 class="wp-block-heading">Branch scoping</h3>



<p class="wp-block-paragraph">GitHub Actions caches are scoped to the branch that created them, plus the default branch as a fallback. A feature branch reads its own caches and the default branch&#8217;s caches. It cannot read a sibling branch&#8217;s cache.</p>



<p class="wp-block-paragraph">The consequence catches a lot of people: if your caching step only runs on pull requests, the default branch never populates a cache, so every new branch starts cold forever. Make sure at least one workflow on the default branch writes it. Runs triggered from fork pull requests get read-only cache access, so those jobs restore but never save.</p>



<h3 class="wp-block-heading">Keys that never hit, or hit and never update</h3>



<p class="wp-block-paragraph">Cache entries are immutable. Once a key exists, writing to it again does nothing. So a key like <code>node-modules-v1</code> gets populated once, on some machine, in some state, and then freezes. A key that hashes the lockfile is correct but creates a brand new entry on every dependency change, which burns through storage.</p>



<p class="wp-block-paragraph">The pattern that survives contact is an exact key plus a prefix fallback: hash the lockfile for the primary key, then use restore keys to fall back to the most recent entry sharing that prefix. You get a partial restore and a fast install instead of a cold one.</p>



<p class="wp-block-paragraph">Then watch the ceiling. Repositories have a default cache storage limit, entries untouched for a week are removed, and once you hit the limit the least recently accessed entries get evicted. Push enough large caches and you get thrashing, where entries are created and evicted at roughly the rate you create them. The tell is a hit rate that looks random. Check the repository&#8217;s cache usage in Actions settings before assuming the config is wrong.</p>



<h3 class="wp-block-heading">Docker layer cache that does not cross machines</h3>



<p class="wp-block-paragraph">On ephemeral runners the local BuildKit cache dies with the machine, so it has to be exported somewhere. Registry-backed cache is the one I reach for first:</p>



<pre class="wp-block-code"><code>docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:${GIT_SHA} 
  --push .</code></pre>



<p class="wp-block-paragraph">Two details do the work. <code>mode=max</code> exports intermediate stage layers, not just the ones that landed in the final image. If the expensive work happens in a builder stage that gets discarded, the default mode caches none of it, which is exactly the case you were trying to fix. Inline cache, the kind that ships alongside the image, only supports the minimal mode, so it cannot help multi-stage builds either.</p>



<p class="wp-block-paragraph">The second detail is a trap. BuildKit cache mounts, the <code>RUN --mount=type=cache</code> kind, are not part of the exported layer cache. They live on the builder. Restoring a registry cache does not restore them, so if the instruction runs again the package manager starts from an empty download cache:</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci</code></pre>



<p class="wp-block-paragraph">That line is excellent on a persistent builder and close to useless on a runner destroyed after every job. Know which one you have before you credit it with a speedup.</p>



<h2 class="wp-block-heading">Pipelines shaped like a staircase</h2>



<p class="wp-block-paragraph">Stages are a scheduling constraint most pipelines inherit by accident. In a stage-based model nothing in the test stage starts until everything in the build stage finishes. Put a forty-second lint job in the same stage as a six-minute compile and your lint feedback arrives six minutes late for no reason.</p>



<p class="wp-block-paragraph">GitLab&#8217;s fix is <code>needs</code>, which lets a job start as soon as its own dependencies finish and ignore stage order entirely:</p>



<pre class="wp-block-code"><code>lint:
  stage: test
  needs: []
  script:
    - make lint

unit:
  stage: test
  needs: ["build"]
  script:
    - make unit</code></pre>



<p class="wp-block-paragraph"><code>needs: []</code> means start immediately, wait for nothing. Lint now reports in under a minute while the compile is still running.</p>



<p class="wp-block-paragraph">GitHub Actions has the opposite default: jobs run in parallel unless you add <code>needs</code>. So the equivalent problem there is over-declared dependencies, usually added early to force a readable graph and never revisited. Go read your <code>needs</code> lines and ask which are real data dependencies and which are narrative ordering.</p>



<p class="wp-block-paragraph">The trade-off is real. A wide graph surfaces every failure at once and costs more compute per run. A narrow graph fails fast and saves minutes, but developers make several round trips to find three problems. For PR pipelines, wide is usually right. For expensive nightly jobs, narrow is.</p>



<h2 class="wp-block-heading">The test suite is not one bottleneck</h2>



<p class="wp-block-paragraph">&#8220;Tests are slow&#8221; gets treated as a single problem. It is usually three, and they want different fixes.</p>



<ul class="wp-block-list">
<li><strong>The suite is serial.</strong> Split by shard across parallel jobs. The cost is per-job setup overhead paid N times, so past a point more shards makes things slower. Find your own break-even instead of assuming more is better.</li>

<li><strong>The suite is flaky.</strong> A retry costs more than one rerun. It costs attention, a context switch, and eventually the team&#8217;s belief that red means anything. Quarantine flaky tests into a non-blocking job with an owner and a deadline. Leaving them in the required path while everyone knows to just re-run is worse than deleting them.</li>

<li><strong>The suite tests the wrong layer.</strong> A test that boots a database and a message broker to assert a validation rule is a unit test wearing a costume. Slowest fix, largest payoff.</li>
</ul>



<p class="wp-block-paragraph">Before touching any of it, get per-test timing out of your test runner and sort descending. Most suites have a handful of tests that dominate, and they are rarely the ones people suspect.</p>



<h2 class="wp-block-heading">The wait that nobody is measuring</h2>



<p class="wp-block-paragraph">Here is the one that costs days rather than minutes, and it appears in no CI metric because from CI&#8217;s point of view nothing is happening.</p>



<p class="wp-block-paragraph">If a workflow is skipped because of path filtering, branch filtering, or a skip directive in the commit message, its checks stay pending. Not skipped. Pending. And if that check is required, the pull request cannot merge. Nothing fails, nothing alerts, the PR just sits there looking almost ready until someone investigates by hand.</p>



<p class="wp-block-paragraph">This bites hardest in monorepos, which is exactly where path filters are most attractive. You add <code>paths</code> to skip the service test suite on documentation-only changes, you save real minutes, and you accidentally build a merge trap.</p>



<p class="wp-block-paragraph">The distinction that matters: a job skipped by an <code>if</code> condition inside a workflow that did run reports success. A workflow that never ran at all reports pending. So either keep the workflow triggering on everything and move the filtering inside, or pair each filtered workflow with a companion using the same name and inverted path filters that does nothing and passes. GitHub documents that second pattern and it works, but it means two files that must stay in sync.</p>



<p class="wp-block-paragraph">The other gate-time offenders are less subtle and just as expensive: one shared staging environment serializing every team, and approvals routed to a single person asleep in another time zone. Plot merge timestamps against last-commit timestamps for a month. The shape of that distribution tells you which problem you have.</p>



<h2 class="wp-block-heading">Troubleshooting: finding your bottleneck in an afternoon</h2>



<ol class="wp-block-list">
<li>Pull the last fifty runs on your default branch and compute queue time and compute time separately. If queue is a meaningful share, fix capacity and cancellation before anything else.</li>

<li>Read your slowest run as a graph, not a list. Wide gaps between a dependency finishing and its dependent starting are either queue or bad ordering.</li>

<li>Compare install duration on a cache hit against a cache miss. If they are similar, the cache is decorative.</li>

<li>Check repository cache storage usage. Thrashing looks like an erratic hit rate with no config changes behind it.</li>

<li>Count retries. The rate of second attempts on the same commit is your flakiness tax, and it is usually higher than anyone guesses.</li>

<li>Measure merge latency, last commit to merge, across a month. The difference between that and pipeline duration is gate time, and that is where the days are.</li>
</ol>



<p class="wp-block-paragraph">Pipeline run data is worth shipping somewhere queryable. Most CI platforms emit webhooks on job completion; feeding those into Prometheus and Grafana, or a hosted platform like Grafana Cloud or Datadog if you would rather not run the storage, turns &#8220;the build feels slow lately&#8221; into a graph you can argue with.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Buying bigger runners before splitting queue time from compute time. The most common wasted spend in CI.</li>

<li>Turning on auto-cancel without setting the per-job interruptible flags, so the setting is technically on and functionally off.</li>

<li>Caching a directory the tool does not read from, then declaring victory because the step went green.</li>

<li>Using the default Docker cache export mode on a multi-stage build and wondering why the builder stage rebuilds every time.</li>

<li>Adding path filters to workflows that back required checks, creating pull requests that can never merge.</li>

<li>Optimizing the pipeline that runs on merge instead of the PR pipeline developers actually wait on.</li>
</ul>



<h2 class="wp-block-heading">Practices that hold up</h2>



<ul class="wp-block-list">
<li>Treat queue time as a first-class metric with its own threshold, not a footnote on build duration.</li>

<li>Cancel superseded runs on branches, never on the default branch and never on deploys.</li>

<li>Make the default branch populate every cache other branches depend on.</li>

<li>Order jobs by how fast they fail, not by how the file reads top to bottom. Cheap checks first, always unblocked.</li>

<li>Give every flaky test an owner and an expiry date, or take it out of the blocking path.</li>

<li>Re-measure after every change and keep the number. Pipelines rot, and the bottleneck moves as soon as you fix the current one.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I find the bottleneck in my CI/CD pipeline?</h3>



<p class="wp-block-paragraph">Split every job into queue time and compute time using the timestamps your platform&#8217;s API already exposes, then compare pipeline duration against real merge latency. Those two comparisons put you in one of three buckets: capacity, build work, or gates. Fixing the wrong bucket is why most optimization efforts show no result.</p>



<h3 class="wp-block-heading">Are self-hosted runners faster than hosted ones?</h3>



<p class="wp-block-paragraph">Per job, often yes, because you pick the hardware and can keep caches warm. Across a working day, frequently no, because a fixed fleet queues under load while a hosted pool does not. The right answer depends on which number is hurting you, compute or queue, which is why measurement comes first. A hybrid setup, self-hosted by default with hosted overflow, avoids committing to either.</p>



<h3 class="wp-block-heading">Why is my cache not speeding anything up?</h3>



<p class="wp-block-paragraph">Usual suspects in order of frequency: the cache is scoped to a branch that never populated it, the key is static so it froze at whatever went in first, the cached directory is not the one the tool reads, or entries are evicted before the next run. Compare install duration on a hit against a miss. Identical duration means the restore is doing nothing regardless of what the log says.</p>



<h3 class="wp-block-heading">Should I parallelize tests or make them faster?</h3>



<p class="wp-block-paragraph">Parallelize first because it is cheap and reversible, then fix the slow tests, because parallelism has a floor. Every shard pays setup cost, so a suite split too many ways spends more time booting containers than running assertions. Sharding buys time to do the real work. It is not the real work.</p>



<h3 class="wp-block-heading">Why does my pull request say pending forever?</h3>



<p class="wp-block-paragraph">Almost certainly a required status check attached to a workflow that was skipped by path filtering, branch filtering, or a commit message directive. A skipped workflow reports pending rather than skipped, so the merge requirement is never satisfied. Either drop the path filter and skip inside the workflow instead, or add a companion workflow with the same name that always passes.</p>



<h3 class="wp-block-heading">Do bigger runners actually help?</h3>



<p class="wp-block-paragraph">They help work that is genuinely CPU or memory bound, such as compilation and container builds. They do nothing for queue time, network-bound dependency downloads, tests waiting on external services, or gate time. Before upgrading, confirm the slow job is saturating the resource you are about to buy more of.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most CI/CD pipeline bottlenecks are waiting, not working. Queue time, cache misses that report as hits, jobs blocked behind stages they do not depend on, and checks that sit pending because a workflow was never triggered. None of them show up on a build duration graph, and all of them are cheap to fix once you can see them. Split the number before you spend anything; the measurement takes an afternoon and usually points at a config change rather than a purchase order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need someone to find where your pipeline is losing time?</h2>



<p class="wp-block-paragraph">This is a large part of what I do as a freelance DevOps engineer. Typical work looks like:</p>



<ul class="wp-block-list">
<li>Instrumenting your pipeline so queue time, compute time and merge latency become three separate numbers you can watch over time</li>

<li>Auditing GitHub Actions and GitLab CI caching to find the restores that report success and deliver nothing</li>

<li>Restructuring stage-based pipelines into dependency graphs so cheap checks report first</li>

<li>Setting up autoscaling self-hosted runners, on Kubernetes with Actions Runner Controller or on plain VMs, with hosted overflow</li>

<li>Fixing Docker build caching for multi-stage images on ephemeral runners, including registry-backed export</li>

<li>Untangling required checks and path filters in monorepos so pull requests stop getting stuck</li>
</ul>



<p class="wp-block-paragraph">Send me a workflow file, a job timeline, or the output of your slowest run, and I will tell you where the minutes are going before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/">CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Terraform Modules Worth Reusing (And the Ones That Cost You)</title>
		<link>https://john-nessime.com/blog/devops/reusable-terraform-modules/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Checkov]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Code Reuse]]></category>
		<category><![CDATA[HCL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Module Versioning]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Terraform Modules]]></category>
		<category><![CDATA[Terraform Registry]]></category>
		<category><![CDATA[Terraform Testing]]></category>
		<category><![CDATA[terraform-docs]]></category>
		<category><![CDATA[tflint]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=198</guid>

					<description><![CDATA[<p>A module's real cost is not writing it, it's upgrading it. Five signals that separate reusable Terraform modules worth adopting from the ones your team will pin at an old version and never touch again, plus the state-address failure that makes version bumps dangerous.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/reusable-terraform-modules/">Terraform Modules Worth Reusing (And the Ones That Cost You)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pull request was tiny. A module version bump, one line, nothing else in the diff. Then the plan came back with something nobody wants to read late on a Thursday: the RDS instance <em>must be replaced</em>.</p>



<p class="wp-block-paragraph">Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource&#8217;s name inside the module. The author had tidied up a <code>resource "aws_db_instance" "this"</code> into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.</p>



<p class="wp-block-paragraph">That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.</p>



<p class="wp-block-paragraph">This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.</p>



<h2 class="wp-block-heading">The bill for a module arrives at upgrade time</h2>



<p class="wp-block-paragraph">Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: <code>module.database.aws_db_instance.this</code>. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.</p>



<p class="wp-block-paragraph">For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.</p>



<p class="wp-block-paragraph">The fix has existed for a long time and it is genuinely good: the <code>moved</code> block. A module author who renames something declares the rename in code, ships it with the release, and every consumer&#8217;s next plan updates state instead of destroying infrastructure.</p>



<pre class="wp-block-code"><code># Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}</code></pre>



<p class="wp-block-paragraph">The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its <code>moved</code> blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.</p>



<p class="wp-block-paragraph">So here is the whole thesis, and everything below is elaboration: <strong>a module is worth reusing if the next version bump is cheap. Everything else is secondary.</strong></p>



<h2 class="wp-block-heading">Five signals that reusable Terraform modules are worth adopting</h2>



<p class="wp-block-paragraph">These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.</p>



<h3 class="wp-block-heading">1. It keeps state addresses stable, or tells you when it does not</h3>



<p class="wp-block-paragraph">Open the repository. Search for <code>moved</code>. Read the changelog for the last major release and see whether renames are called out.</p>



<p class="wp-block-paragraph">A module with no <code>moved</code> blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says &#8220;v3.0 renames the launch template resource, here are the moved blocks&#8221; is telling you it has thought about consumers.</p>



<h3 class="wp-block-heading">2. The input surface is narrow and typed</h3>



<p class="wp-block-paragraph">Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.</p>



<p class="wp-block-paragraph">Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.</p>



<pre class="wp-block-code"><code>variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}</code></pre>



<p class="wp-block-paragraph">Three things are doing work there. <code>optional()</code> with a default means a caller who wants the simple case writes two lines instead of six. <code>nullable = false</code> means a caller cannot pass <code>null</code> and get a confusing failure three resources deep. The <code>validation</code> block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.</p>



<p class="wp-block-paragraph">That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.</p>



<h3 class="wp-block-heading">3. You can pin it, and the pin means something</h3>



<p class="wp-block-paragraph">This trips people up because the two common source types behave differently.</p>



<pre class="wp-block-code"><code># Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~&gt; 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}</code></pre>



<p class="wp-block-paragraph">The <code>~&gt;</code> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as <code>~&gt; 5.0.1</code> it accepts patches only. Which one you want depends entirely on whether you trust the author&#8217;s reading of semantic versioning, and the honest answer for third-party modules is usually &#8220;not enough to auto-accept minors in production&#8221;.</p>



<p class="wp-block-paragraph">For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at <code>init</code> time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.</p>



<h3 class="wp-block-heading">4. It has tests you can run without a cloud account</h3>



<p class="wp-block-paragraph">Terraform&#8217;s built-in test framework uses <code>.tftest.hcl</code> files and the same HCL you already write. The critical detail for module authors is that a run block set to <code>command = plan</code> creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.</p>



<pre class="wp-block-code"><code># tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}</code></pre>



<p class="wp-block-paragraph">Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.</p>



<p class="wp-block-paragraph">OpenTofu runs the same format through <code>tofu test</code>, and additionally recognises a <code>.tofutest.hcl</code> extension for tests that should only run there. Useful if you maintain modules that target both tools.</p>



<h3 class="wp-block-heading">5. Somebody answers issues</h3>



<p class="wp-block-paragraph">Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.</p>



<p class="wp-block-paragraph">Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when <code>init</code> refuses to resolve a version constraint.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Three kinds of module, and which ones actually pay off</h2>



<p class="wp-block-paragraph">Not everything that <em>can</em> be a module <em>should</em> be. It helps to name the three shapes you see in the wild.</p>



<h3 class="wp-block-heading">The thin wrapper</h3>



<p class="wp-block-paragraph">One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.</p>



<p class="wp-block-paragraph">There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.</p>



<h3 class="wp-block-heading">The composition module</h3>



<p class="wp-block-paragraph">Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.</p>



<p class="wp-block-paragraph">Most modules worth reusing live here.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.</p>



<h2 class="wp-block-heading">Public modules or your own?</h2>



<p class="wp-block-paragraph">The case for public modules is stronger than the &#8220;we have specific needs&#8221; crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.</p>



<p class="wp-block-paragraph">The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.</p>



<p class="wp-block-paragraph">My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.</p>



<p class="wp-block-paragraph">There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp&#8217;s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.</p>



<p class="wp-block-paragraph">For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.</p>



<h2 class="wp-block-heading">Making your own modules worth reusing</h2>



<p class="wp-block-paragraph">If you are on the authoring side, this is the order I would do things in.</p>



<ol class="wp-block-list">
<li><strong>Write it inline first, twice.</strong> Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module&#8217;s interface, discovered rather than guessed.</li>



<li><strong>Extract with <code>moved</code> blocks in the same commit.</strong> Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.</li>



<li><strong>Type the inputs properly.</strong> Objects with <code>optional()</code> defaults over flat strings, <code>nullable = false</code> where null makes no sense, <code>validation</code> blocks with error messages that name the fix.</li>



<li><strong>Prefer <code>for_each</code> over <code>count</code> for anything collection-shaped.</strong> With <code>count</code>, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.</li>



<li><strong>Generate the docs.</strong> <code>terraform-docs</code> reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.</li>



<li><strong>Lint and scan on every PR.</strong> <code>terraform fmt -check</code>, <code>terraform validate</code>, <code>tflint</code> for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.</li>



<li><strong>Tag releases, and mean it.</strong> A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.</li>
</ol>



<p class="wp-block-paragraph">One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.</p>



<h2 class="wp-block-heading">Troubleshooting the plans that scare you</h2>



<h3 class="wp-block-heading">&#8220;Forces replacement&#8221; after a version bump</h3>



<p class="wp-block-paragraph">Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.</p>



<pre class="wp-block-code"><code>terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'</code></pre>



<p class="wp-block-paragraph">If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version&#8217;s source. A rename with no <code>moved</code> block is the usual answer, and you can write the <code>moved</code> block yourself in your root configuration as a workaround.</p>



<h3 class="wp-block-heading">A module with a provider block inside it</h3>



<p class="wp-block-paragraph">Modules should declare which providers they need, not configure them. A module that contains a configured <code>provider</code> block rather than just a <code>required_providers</code> entry limits what callers can do with it, including using <code>for_each</code> or <code>count</code> on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.</p>



<h3 class="wp-block-heading">Changes to the module source are not picked up</h3>



<p class="wp-block-paragraph">Remote modules are downloaded into the working directory during <code>init</code>. Changing a <code>ref</code> or a <code>version</code> constraint in your configuration does not re-download anything on its own. Run <code>terraform init -upgrade</code>. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.</p>



<h3 class="wp-block-heading">Registry version list looks stale</h3>



<p class="wp-block-paragraph">A registry&#8217;s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry&#8217;s view after any change to that integration.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Pinning to a branch.</strong> <code>?ref=main</code> is not a pin. It is an unannounced deployment scheduled for whenever someone next runs <code>init</code>.</li>



<li><strong>Modularising after one use.</strong> You end up encoding the first use case&#8217;s assumptions as the abstraction, then bolting flags onto it for every case after.</li>



<li><strong>Copying environments instead of parameterising them.</strong> Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.</li>



<li><strong>Passing every provider argument straight through.</strong> If a module&#8217;s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.</li>



<li><strong>Deleting <code>moved</code> blocks too early.</strong> They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.</li>



<li><strong>Applying a plan you did not read.</strong> Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.</li>
</ul>



<h2 class="wp-block-heading">How I decide whether to adopt a module</h2>



<p class="wp-block-paragraph">Roughly ten minutes, in this order:</p>



<ol class="wp-block-list">
<li>Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.</li>



<li>Grep for <code>moved</code>. Its presence tells me the author thinks about consumers.</li>



<li>Read the last major release notes. Are breaking changes explained, with an upgrade path?</li>



<li>Check for a <code>tests</code> directory and whether CI runs on pull requests.</li>



<li>Look at the issue tracker&#8217;s recent activity, not its total count.</li>



<li>Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.</li>
</ol>



<p class="wp-block-paragraph">And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How many input variables is too many for a Terraform module?</h3>



<p class="wp-block-paragraph">There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.</p>



<h3 class="wp-block-heading">Should I use public modules or write my own?</h3>



<p class="wp-block-paragraph">Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else&#8217;s.</p>



<h3 class="wp-block-heading">Do Terraform modules work with OpenTofu?</h3>



<p class="wp-block-paragraph">Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.</p>



<h3 class="wp-block-heading">How do I pin a module version from a Git source?</h3>



<p class="wp-block-paragraph">The <code>version</code> argument only applies to registry sources. For Git, append a <code>ref</code> query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at <code>init</code> time, which defeats the purpose of pinning entirely.</p>



<h3 class="wp-block-heading">What happens if a module removes a resource in a new version?</h3>



<p class="wp-block-paragraph">Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.</p>



<h3 class="wp-block-heading">Can I test a Terraform module without cloud credentials?</h3>



<p class="wp-block-paragraph">Yes, for a useful subset. Run blocks set to <code>command = plan</code> evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.</p>



<h3 class="wp-block-heading">Do I need a private registry to share modules internally?</h3>



<p class="wp-block-paragraph">No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.</p>



<p class="wp-block-paragraph">The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your module library?</h2>



<p class="wp-block-paragraph">I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:</p>



<ul class="wp-block-list">
<li>Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations</li>



<li>Adding <code>moved</code> blocks and upgrade notes to in-house modules so version bumps stop being incidents</li>



<li>Narrowing wide module interfaces with typed objects, <code>optional()</code> defaults and validation rules that fail with useful messages</li>



<li>Setting up module CI: <code>fmt</code>, <code>validate</code>, <code>tflint</code>, policy scanning and generated docs on every pull request</li>



<li>Writing <code>terraform test</code> suites that run on plan, including negative tests that prove your validation rules actually fire</li>



<li>Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all</li>
</ul>



<p class="wp-block-paragraph">Send me a module&#8217;s <code>variables.tf</code> and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/reusable-terraform-modules/">Terraform Modules Worth Reusing (And the Ones That Cost You)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</title>
		<link>https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 10:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[GitLab CI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Jenkins]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=201</guid>

					<description><![CDATA[<p>A practical comparison of GitLab CI, GitHub Actions and Jenkins that skips the feature table. What actually decides the choice is where your code lives, whether the runner can reach the deploy target, and who owns the control plane at 2am. Includes the cost mechanics, the security failure modes, and a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question almost always arrives the same way. Someone pastes a long Jenkinsfile into a channel, adds a shrug emoji, and asks how hard it would be to move this to GitHub Actions. Nobody in the thread is asking the questions that will actually decide the outcome, because those questions are boring and the YAML is right there looking portable.</p>



<p class="wp-block-paragraph">Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.</p>



<p class="wp-block-paragraph">This post compares <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest &#8220;where it wins and where it doesn&#8217;t&#8221;, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.</p>



<h2 class="wp-block-heading">The part of the decision that actually bites</h2>



<p class="wp-block-paragraph">Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.</p>



<p class="wp-block-paragraph">Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.</p>



<p class="wp-block-paragraph">The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system &#8220;for now&#8221;, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.</p>



<p class="wp-block-paragraph">So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the &#8220;below the line&#8221; half cheapest for your specific situation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What each platform is really optimising for</h2>



<p class="wp-block-paragraph">Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.</p>



<h3 class="wp-block-heading">GitHub Actions: shortest path from push to green</h3>



<p class="wp-block-paragraph">Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else&#8217;s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.</p>



<p class="wp-block-paragraph">A minimal hardened job header looks like this, and every line in it is there for a reason:</p>



<pre class="wp-block-code"><code>name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"</code></pre>



<p class="wp-block-paragraph">The <code>permissions</code> block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The <code>concurrency</code> block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.</p>



<h3 class="wp-block-heading">GitLab CI: one platform owning the whole lifecycle</h3>



<p class="wp-block-paragraph">GitLab&#8217;s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.</p>



<p class="wp-block-paragraph">The pipeline language rewards you for using its graph features rather than treating stages as a queue:</p>



<pre class="wp-block-code"><code>workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh</code></pre>



<p class="wp-block-paragraph">The <code>workflow:rules</code> block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. <code>needs</code> turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. <code>interruptible: true</code> makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project&#8217;s CI settings; the keyword alone does nothing. And <code>expire_in</code> is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.</p>



<h3 class="wp-block-heading">Jenkins: whatever you need, on hardware you control</h3>



<p class="wp-block-paragraph">Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor&#8217;s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.</p>



<pre class="wp-block-code"><code>pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things in that file are worth pointing at. <code>timeout</code> and <code>buildDiscarder</code> are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And <code>timestamps()</code> and <code>cleanWs()</code> both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">GitLab CI vs GitHub Actions vs Jenkins on cost</h2>



<p class="wp-block-paragraph">Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.</p>



<ul class="wp-block-list"><li><strong>GitHub Actions.</strong> Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.</li><li><strong>GitLab CI.</strong> Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why &#8220;pay for seats, run your own runners&#8221; is such a common GitLab shape.</li><li><strong>Jenkins.</strong> No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.</li></ul>



<p class="wp-block-paragraph">Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.</p>



<p class="wp-block-paragraph">Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Security is where the differences stop being cosmetic</h2>



<p class="wp-block-paragraph">Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.</p>



<ul class="wp-block-list"><li><strong>Untrusted pull requests.</strong> This is the classic GitHub Actions trap. The <code>pull_request</code> trigger runs fork code without access to your secrets. The <code>pull_request_target</code> trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request&#8217;s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub&#8217;s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.</li><li><strong>Expression injection.</strong> Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.</li><li><strong>Self-hosted runners on public repositories.</strong> Don&#8217;t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.</li><li><strong>Mutable action and image references.</strong> Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.</li><li><strong>Credential scope.</strong> Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.</li><li><strong>Jenkins specifically.</strong> The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project&#8217;s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run in an afternoon</h2>



<ol class="wp-block-list"><li><strong>Follow platform gravity first.</strong> Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.</li><li><strong>Check whether a runner can reach the deploy target.</strong> Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.</li><li><strong>Name the on-call owner.</strong> Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.</li><li><strong>Model the compute, not the sticker price.</strong> Take last month&#8217;s build minutes, split them by operating system and machine size, and apply each platform&#8217;s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.</li><li><strong>Inventory what the plugins actually do.</strong> If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The &#8220;needs writing&#8221; pile is your real migration estimate.</li><li><strong>Trial with your ugliest pipeline.</strong> Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.</li></ol>



<h2 class="wp-block-heading">The hybrid pattern nobody puts on the slide</h2>



<p class="wp-block-paragraph">A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;Jenkins is legacy.&#8221;</strong> It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.</li><li><strong>&#8220;We&#8217;ll save money with self-hosted runners.&#8221;</strong> On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.</li><li><strong>&#8220;GitLab CI is just GitHub Actions with different YAML.&#8221;</strong> The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.</li><li><strong>&#8220;We&#8217;ll stay vendor-neutral by keeping all logic in shell scripts.&#8221;</strong> Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.</li><li><strong>&#8220;The bigger marketplace wins.&#8221;</strong> Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ul class="wp-block-list"><li>Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.</li><li>Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.</li><li>Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.</li><li>Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.</li><li>Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Jenkins still worth using?</h3>



<p class="wp-block-paragraph">Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.</p>



<h3 class="wp-block-heading">Which is cheaper, GitLab CI or GitHub Actions?</h3>



<p class="wp-block-paragraph">It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab&#8217;s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.</p>



<h3 class="wp-block-heading">Can I use GitLab CI with a GitHub repository?</h3>



<p class="wp-block-paragraph">There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.</p>



<h3 class="wp-block-heading">How long does migrating from Jenkins to GitHub Actions or GitLab CI take?</h3>



<p class="wp-block-paragraph">Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.</p>



<h3 class="wp-block-heading">Are self-hosted runners safe?</h3>



<p class="wp-block-paragraph">They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.</p>



<h3 class="wp-block-heading">Do I need to pin GitHub Actions to a commit SHA?</h3>



<p class="wp-block-paragraph">For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.</p>



<h3 class="wp-block-heading">Is it reasonable to run more than one CI system?</h3>



<p class="wp-block-paragraph">Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Comparing <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.</p>



<p class="wp-block-paragraph">Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.</p>



<h2 class="wp-block-heading">Need help choosing or moving a CI pipeline?</h2>



<p class="wp-block-paragraph">This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:</p>



<ul class="wp-block-list"><li>A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.</li><li>Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.</li><li>Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.</li><li>Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.</li><li>Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.</li><li>CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.</li><li>Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.</li></ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Cluster Becomes the Product: When Kubernetes Is Overkill</title>
		<link>https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 15:26:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=93</guid>

					<description><![CDATA[<p>Three hours into an ingress controller upgrade for an application that is one Go binary and a Postgres database. Nothing is broken. The cluster has just become the thing you operate. An honest look at what Kubernetes costs, when it earns that cost, and what to run instead.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/">The Cluster Becomes the Product: When Kubernetes Is Overkill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It&#8217;s a Tuesday and you&#8217;re three hours into an ingress controller upgrade. The CRD schema changed, the annotations you copied from a blog post two years ago are deprecated, and cert-manager wants a newer API version than the one your Helm chart pins.</p>



<p class="wp-block-paragraph">The application behind all of this is one Go binary and a Postgres database. It serves a few hundred requests a minute. Nobody has touched its code in a fortnight.</p>



<p class="wp-block-paragraph">That is the moment worth noticing: <strong>when Kubernetes is overkill</strong>, the tell is not that anything is broken. Everything works. It is that the cluster has quietly become the thing you operate, and the product has become a tenant.</p>



<h2 class="wp-block-heading">What it actually buys you</h2>



<p class="wp-block-paragraph">Worth being fair, because the case for Kubernetes is real and I reach for it regularly.</p>



<p class="wp-block-paragraph">It gives you bin-packing across many workloads, so a fleet of services shares machines efficiently. It gives you a declarative API that a platform team can build on, so twelve product teams deploy without twelve bespoke pipelines. It gives you self-healing, rolling updates, horizontal autoscaling and namespace-level isolation as defaults rather than as things you script. And it gives you a portable vocabulary: an engineer who knows Kubernetes knows most of your infrastructure on day one.</p>



<p class="wp-block-paragraph">Every one of those benefits scales with the number of services and the number of teams. None of them scale with how modern you want to feel.</p>



<h2 class="wp-block-heading">The bill nobody itemises</h2>



<p class="wp-block-paragraph">The cost is not the control plane fee. It is the surface area you have agreed to keep current.</p>



<p class="wp-block-paragraph">Kubernetes ships three minor releases a year and supports each for about fourteen months, and you can skip at most one minor version when upgrading. So there is a permanent, non-negotiable upgrade cadence: roughly every four months something needs planning, testing and a maintenance window, forever. Fall behind and you are doing multi-hop upgrades on an unsupported version, which is the worst combination available.</p>



<p class="wp-block-paragraph">Underneath that sit the components nobody counts when they say &#8220;we&#8217;ll just run Kubernetes&#8221;: a CNI plugin, a CSI driver, an ingress controller, cert-manager, external-dns, a metrics pipeline, an autoscaler, RBAC, and whatever operators your databases need. Each has its own release cycle, its own breaking changes and its own opinions about the others. That is where the Tuesday goes.</p>



<p class="wp-block-paragraph">Then there is debugging. A request that used to fail in one place now fails in one of several, and telling apart an application bug from a NetworkPolicy, a readiness probe, a DNS caching issue or a node under memory pressure is a genuine skill. On a good team that skill exists. On a three-person team it exists in one person&#8217;s head and goes on holiday with them.</p>



<h2 class="wp-block-heading">Signals you don&#8217;t need it yet</h2>



<ul class="wp-block-list">
<li>You have fewer than about five services, and most of them are one binary and a database.</li>
<li>One team owns everything, so there is nothing to isolate from anyone.</li>
<li>Traffic is predictable enough that you size for peak and stop thinking about it.</li>
<li>Nobody on the team has run a cluster in production before.</li>
<li>Your deploys are already fine. Nobody is complaining about them.</li>
<li>You cannot name the specific problem Kubernetes is solving, only the general one.</li>
<li>The cluster would run on two nodes, which means you have a scheduler for a decision with two possible answers.</li>
</ul>



<h2 class="wp-block-heading">Signals you probably do</h2>



<ul class="wp-block-list">
<li>Multiple teams deploying independently and tripping over each other&#8217;s environments.</li>
<li>Enough services that per-service deployment scripts have become their own maintenance problem.</li>
<li>Genuinely variable load where autoscaling saves real money, not hypothetical money.</li>
<li>A compliance or tenancy requirement that maps cleanly onto namespaces and network policy.</li>
<li>You are already running an operator-based product, like a database platform, that expects Kubernetes.</li>
<li>Somebody&#8217;s whole job is the platform, and it is not also somebody&#8217;s whole job to ship features.</li>
</ul>



<p class="wp-block-paragraph">That last one is the honest test. Kubernetes is not expensive to install. It is expensive to own, and ownership needs a name attached to it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What I&#8217;d reach for instead</h2>



<p class="wp-block-paragraph">Roughly in order of how much you take on.</p>



<p class="wp-block-paragraph"><strong>One server, systemd, and a reverse proxy.</strong> Underrated to the point of being contrarian, and completely adequate for a large number of real businesses. A VPS from Hetzner, InterServer or DigitalOcean, nginx or Caddy in front, systemd units for your services, automated backups off the box. Restart policies and health checks are built in; you are not going without them, you are just getting them from init instead of from a control loop.</p>



<p class="wp-block-paragraph"><strong>Docker Compose on one box.</strong> The step up when you want containers without a scheduler. The entire deployment is a file you can read in one screen:</p>



<pre class="wp-block-code"><code>services:
  app:
    image: ghcr.io/acme/api:1.4.2      # pinned, deployed by changing this line
    restart: always
    ports: ["127.0.0.1:8080:8080"]     # proxy handles TLS and the public port
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16                 # pin the major, upgrade deliberately
    restart: always
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes:
  pgdata:</code></pre>



<p class="wp-block-paragraph">The honest limitation: one box is one box. Losing it is an outage, and a redeploy is a brief gap unless you put something in front. For plenty of internal tools and early products, that is an acceptable trade stated out loud rather than an oversight.</p>



<p class="wp-block-paragraph"><strong>A platform that runs containers for you.</strong> Cloud Run, App Runner, Fly.io, Render, Railway. You hand over a container image and get scaling, TLS, rolling deploys and health checks without operating any of it. This is the option most teams should look at hardest, because it buys back the majority of what Kubernetes offers at almost none of the operational cost. You pay in per-unit pricing and in reduced control, and both are usually worth it below a certain scale.</p>



<p class="wp-block-paragraph"><strong>ECS on Fargate, or Nomad.</strong> Real schedulers with a fraction of the concepts. If you need orchestration but not the ecosystem, these are legitimate destinations rather than waypoints on the road to Kubernetes.</p>



<p class="wp-block-paragraph">And if you have decided Kubernetes genuinely is the answer, use a managed control plane, and consider a lightweight distribution such as k3s for small or edge deployments. Do not build the cluster yourself to save money. That is the most expensive saving in infrastructure.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><em>&#8220;We&#8217;ll need it eventually.&#8221;</em> Possibly. Adopting it three years early costs three years of upgrade cycles to buy an option you may not exercise, and the migration is not much harder later than it is now.</p>



<p class="wp-block-paragraph"><em>&#8220;It&#8217;s the industry standard.&#8221;</em> For orchestrating fleets, yes. Most applications are not fleets. Standard does not mean universally appropriate, and the same argument would put a load balancer in front of a single server.</p>



<p class="wp-block-paragraph"><em>&#8220;It avoids vendor lock-in.&#8221;</em> Partly true and frequently oversold. You have swapped a cloud provider&#8217;s API for a distribution, a set of controllers, and a pile of YAML that encodes assumptions about all of them. Portability lives in your container images and your data, not in the scheduler.</p>



<p class="wp-block-paragraph"><em>&#8220;It&#8217;ll help us hire.&#8221;</em> It helps you hire people who want to run Kubernetes. Whether that is the same as the people who will build your product is worth a moment&#8217;s thought.</p>



<h2 class="wp-block-heading">You&#8217;re not painting yourself into a corner</h2>



<p class="wp-block-paragraph">This is the part that makes the decision easy to defer, and it is the most useful thing in this post.</p>



<p class="wp-block-paragraph">The durable investments are containerising your application, externalising configuration into environment variables, keeping state out of the application and in a database or object storage, exposing a health endpoint, and logging to stdout. Do those and your app already satisfies most of what a Kubernetes manifest asks for. The scheduler underneath is genuinely swappable.</p>



<p class="wp-block-paragraph">So the choice is not Kubernetes now or a painful rewrite later. It is Kubernetes now, or the same containers running somewhere simpler until the complexity is earned.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How many services before Kubernetes makes sense?</h3>



<p class="wp-block-paragraph">There is no clean number, and anyone offering one is guessing. The better question is how many teams deploy independently. One team can coordinate without a platform. Four cannot, and that is when the abstraction starts paying for itself.</p>



<h3 class="wp-block-heading">Is Docker Compose really production-ready?</h3>



<p class="wp-block-paragraph">On a single host, for a workload that tolerates brief restarts, yes. It runs plenty of real systems. What it does not give you is failover across machines, so be clear that losing the host means an outage and decide whether that is acceptable rather than discovering it.</p>



<h3 class="wp-block-heading">Doesn&#8217;t managed Kubernetes remove the operational burden?</h3>



<p class="wp-block-paragraph">It removes the control plane, which is the part that was never your main problem. Ingress, storage, networking, upgrades and the add-on ecosystem all remain yours. Managed offerings genuinely help, and they narrow the gap rather than closing it.</p>



<h3 class="wp-block-heading">What about autoscaling? I can&#8217;t get that elsewhere.</h3>



<p class="wp-block-paragraph">You can. Cloud Run, App Runner, Fly and ECS all autoscale, several of them to zero, with no cluster to maintain. Autoscaling is a common reason to reach for Kubernetes and one of the weakest, because it is the feature most thoroughly commoditised elsewhere.</p>



<h3 class="wp-block-heading">How hard is it to migrate later?</h3>



<p class="wp-block-paragraph">Much easier than migrating off it. If your app is containerised, stateless, configured by environment and observable, moving to Kubernetes is mostly writing manifests. The hard parts of that migration are the ones you should be fixing anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Kubernetes is not complicated by accident. It is complicated because it solves a genuinely complicated problem, and if you do not have that problem you are paying the complexity and collecting none of the benefit.</p>



<p class="wp-block-paragraph">So make it a decision with a reason attached. Write down the specific thing you cannot do today, and check whether a cluster is the cheapest way to fix it. Sometimes it is. Often the honest answer is one server, a reverse proxy, and an afternoon spent on backups instead.</p>



<h2 class="wp-block-heading">Trying to decide?</h2>



<p class="wp-block-paragraph">I get called in on both sides of this: teams drowning in a cluster they did not need, and teams who genuinely need one and are putting it off. Work I take on:</p>



<ul class="wp-block-list">
<li>A straight assessment of whether Kubernetes fits your team and workload, with the reasoning written down rather than an opinion delivered.</li>
<li>Building the simpler thing properly: single-host or Compose deployments with TLS, health checks, deploys and backups that hold up in production.</li>
<li>Moving off an over-provisioned cluster onto something proportionate, without downtime.</li>
<li>Getting an application genuinely portable so the decision stays reversible: containerised, stateless, configured by environment, observable.</li>
<li>Setting up managed Kubernetes properly when it is the right answer, including the upgrade process nobody plans for.</li>
<li>Reviewing the cost, in money and in engineering hours, of what you currently run.</li>
</ul>



<p class="wp-block-paragraph">Tell me how many services you run and how many people deploy them, and I will tell you what I would build.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/when-kubernetes-is-overkill/">The Cluster Becomes the Product: When Kubernetes Is Overkill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
