<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Rolling Updates | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/rolling-updates/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/rolling-updates/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 04 Oct 2026 06:06:28 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Rolling Updates | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/rolling-updates/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</title>
		<link>https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 04 Oct 2026 06:06:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Amazon EKS]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Blue/Green Deployments]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cluster Autoscaler]]></category>
		<category><![CDATA[Cluster Insights]]></category>
		<category><![CDATA[CoreDNS]]></category>
		<category><![CDATA[Deprecated APIs]]></category>
		<category><![CDATA[Drift Detection]]></category>
		<category><![CDATA[EKS Auto Mode]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Karpenter]]></category>
		<category><![CDATA[Node Autoscaling]]></category>
		<category><![CDATA[Node Lifecycle]]></category>
		<category><![CDATA[Pod Disruption Budgets]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[Runbooks]]></category>
		<category><![CDATA[VPC CNI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=839</guid>

					<description><![CDATA[<p>The control plane bump is the part AWS handles and almost never breaks. The damage happens in the data plane rollout afterwards. A practical walk through the three failure families that actually stall EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the node drain, plus what each escape hatch really costs.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The control plane upgrade finished in about twenty minutes and nobody noticed. Then the managed node group update went to <code>Updating</code>, sat there, and came back with <code>PodEvictionFailure</code>. Half the group was on the new AMI and half on the old, and the thing blocking the drain was a PodDisruptionBudget somebody wrote eighteen months earlier for a two-replica deployment.</p>



<p class="wp-block-paragraph">That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.</p>



<p class="wp-block-paragraph">This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.</p>



<h2 class="wp-block-heading">Why the control plane is the easy half</h2>



<p class="wp-block-paragraph">An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.</p>



<p class="wp-block-paragraph">Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the API calls you did not know you were making</h2>



<p class="wp-block-paragraph">Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.</p>



<p class="wp-block-paragraph">Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.</p>



<h3 class="wp-block-heading">Cluster insights, and what it is actually reading</h3>



<pre class="wp-block-code"><code>aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111</code></pre>



<p class="wp-block-paragraph">EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in <code>ERROR</code> until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:</p>



<pre class="wp-block-code"><code>aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster</code></pre>



<p class="wp-block-paragraph">The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.</p>



<h3 class="wp-block-heading">Ask the API server directly</h3>



<p class="wp-block-paragraph">The API server keeps its own counter, and it names the release that removes each API. Fastest check available:</p>



<pre class="wp-block-code"><code>kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis</code></pre>



<p class="wp-block-paragraph">Read the <code>removed_release</code> label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.</p>



<p class="wp-block-paragraph">For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:</p>



<pre class="wp-block-code"><code>fields @message
| filter `annotations.k8s.io/deprecated` = "true"</code></pre>



<p class="wp-block-paragraph">The <code>user.username</code> field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart&#8217;s owner. Static scanners like <code>kubent</code> and <code>pluto</code> cover the other half, the manifests that have not run recently. Use both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: add-ons and the conflict resolution you picked once</h2>



<p class="wp-block-paragraph">Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.</p>



<p class="wp-block-paragraph">When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:</p>



<ul class="wp-block-list">
<li><code>NONE</code> leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.</li>



<li><code>OVERWRITE</code> reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.</li>



<li><code>PRESERVE</code> keeps your value through the update.</li>
</ul>



<p class="wp-block-paragraph"><code>OVERWRITE</code> is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with <code>OVERWRITE</code> will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.</p>



<pre class="wp-block-code"><code>aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version &lt;compatible-version&gt; 
    --resolve-conflicts PRESERVE</code></pre>



<p class="wp-block-paragraph">One more trap: the same value means different things on create and on update. On create, <code>PRESERVE</code> behaves close to <code>NONE</code>, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.</p>



<p class="wp-block-paragraph">The durable fix is to stop editing add-on resources with <code>kubectl</code> at all. Put customizations in the add-on&#8217;s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: PDB deadlocks and the fifteen-minute drain clock</h2>



<p class="wp-block-paragraph">This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:</p>



<ol class="wp-block-list">
<li>Pick a node to replace, up to the configured <code>maxUnavailable</code>.</li>



<li>Cordon it once a new node reaches <code>Ready</code>.</li>



<li>Drain the pods off it.</li>



<li>Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.</li>



<li>Repeat until nothing is left on the old launch template version.</li>
</ol>



<p class="wp-block-paragraph">Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with <code>PodEvictionFailure</code> and you are left with a half-migrated node group.</p>



<h3 class="wp-block-heading">The deadlock nobody sees coming</h3>



<p class="wp-block-paragraph">A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.</p>



<p class="wp-block-paragraph">One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:</p>



<pre class="wp-block-code"><code>kubectl get pdb --all-namespaces</code></pre>



<p class="wp-block-paragraph">Read the <code>ALLOWED DISRUPTIONS</code> column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.</p>



<p class="wp-block-paragraph">There is a second cause of <code>PodEvictionFailure</code> with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates <code>NoSchedule</code> unconditionally.</p>



<h3 class="wp-block-heading">What the force flag actually costs</h3>



<pre class="wp-block-code"><code>aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force</code></pre>



<p class="wp-block-paragraph">Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.</p>



<p class="wp-block-paragraph">My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.</p>



<h3 class="wp-block-heading">If you run Karpenter, the levers are different</h3>



<p class="wp-block-paragraph">Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node&#8217;s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:</p>



<pre class="wp-block-code"><code>spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h</code></pre>



<p class="wp-block-paragraph">Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not &#8220;no disruption&#8221;, they are &#8220;unlimited disruption&#8221;.</p>



<p class="wp-block-paragraph">The other lever, <code>terminationGracePeriod</code> on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the <code>karpenter.sh/do-not-disrupt</code> annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The rollback window changes the calculation</h2>



<p class="wp-block-paragraph">Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.</p>



<p class="wp-block-paragraph">What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.</p>



<p class="wp-block-paragraph">It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.</p>



<h2 class="wp-block-heading">Troubleshooting a stalled or failed upgrade</h2>



<p class="wp-block-paragraph">When a node group update fails, the reason comes back as an error code. Start there rather than guessing:</p>



<pre class="wp-block-code"><code>aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup</code></pre>



<ul class="wp-block-list">
<li><strong>PodEvictionFailure.</strong> A budget blocked the drain, or something tolerates all taints. Check <code>ALLOWED DISRUPTIONS</code> first, then tolerations.</li>



<li><strong>NodeCreationFailure in scale up.</strong> New nodes never reached <code>Ready</code>. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.</li>



<li><strong>Slow bootstrap.</strong> A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.</li>



<li><strong>Pods stuck terminating with no budget in sight.</strong> Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.</li>



<li><strong>Insights still in ERROR after the fix.</strong> The thirty day audit window. Refresh, confirm another way, move on.</li>
</ul>



<p class="wp-block-paragraph">One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.</li>



<li>Reaching for <code>OVERWRITE</code> to make a failing add-on update succeed. It succeeds by discarding your configuration.</li>



<li>Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.</li>



<li>Leaving <code>maxUnavailable</code> at its default of one on a large node group, then wondering why the rollout takes hours.</li>



<li>Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.</li>
</ul>



<h2 class="wp-block-heading">How I sequence EKS version upgrades</h2>



<ol class="wp-block-list">
<li>Read the release notes for the target version. Not the summary post, the removals section.</li>



<li>Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.</li>



<li>Audit PDBs and fix every zero. Highest-value ten minutes in the process.</li>



<li>Update add-ons with <code>PRESERVE</code> to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.</li>



<li>Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.</li>



<li>Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.</li>



<li>Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.</li>
</ol>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can you do EKS version upgrades with zero downtime?</h3>



<p class="wp-block-paragraph">For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.</p>



<h3 class="wp-block-heading">Why does the update fail with PodEvictionFailure when my PDBs look fine?</h3>



<p class="wp-block-paragraph">Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check <code>ALLOWED DISRUPTIONS</code> across all namespaces, then check tolerations.</p>



<h3 class="wp-block-heading">Is the force update option safe?</h3>



<p class="wp-block-paragraph">Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.</p>



<h3 class="wp-block-heading">Does Karpenter make upgrades easier than managed node groups?</h3>



<p class="wp-block-paragraph">More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with <code>kubectl</code> rather than through its configuration values, a chart pinned before an API was removed.</p>



<p class="wp-block-paragraph">Which means the pre-flight checks are the upgrade. Ten minutes reading <code>ALLOWED DISRUPTIONS</code> and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your upgrade path?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:</p>



<ul class="wp-block-list">
<li>Audit every PDB against its replica count and flag the ones that will deadlock a drain.</li>



<li>Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.</li>



<li>Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.</li>



<li>Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.</li>



<li>Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.</li>



<li>Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.</li>
</ul>



<p class="wp-block-paragraph">If you are staring at a failed node group update right now, send me the output of <code>kubectl get pdb --all-namespaces</code> and the error code from <code>describe-nodegroup</code>. That is usually enough to say what is holding the drain.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</title>
		<link>https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 11 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Health Checks]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=189</guid>

					<description><![CDATA[<p>Readiness probes are necessary for Kubernetes zero-downtime rollouts, but they only close one of four gaps. Here is what actually drops requests during a deploy: the race between SIGTERM and endpoint propagation, probes that lie, surge settings that quietly delete capacity, and load balancers that never watched EndpointSlices in the first place.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The rollout goes green. <code>kubectl rollout status</code> exits zero, every pod reports <code>1/1 Running</code>, and the deploy channel gets the thumbs-up emoji. Then someone from support pastes a screenshot: a spike of 502s, about forty seconds wide, lined up exactly with the deploy. Nobody paged. Nothing restarted. The graph just has a small cliff in it.</p>



<p class="wp-block-paragraph">That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that <strong>Kubernetes zero-downtime rollouts</strong> depend on, and it is not the question that usually bites.</p>



<p class="wp-block-paragraph">This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.</p>



<h2 class="wp-block-heading">What a zero-downtime rollout actually requires</h2>



<p class="wp-block-paragraph">Three independent things have to be true at the same time, and each is owned by a different part of the system.</p>



<ul class="wp-block-list">
<li><strong>Capacity never dips below what traffic needs.</strong> Owned by the Deployment&#8217;s rolling update strategy.</li>

<li><strong>No pod receives traffic before it can serve it.</strong> Owned by the readiness probe.</li>

<li><strong>No pod receives traffic after it stops serving it.</strong> Owned by graceful termination, and this is the one nobody configures.</li>
</ul>



<p class="wp-block-paragraph">Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.</p>



<h2 class="wp-block-heading">Failure family one: traffic arrives before the pod can serve it</h2>



<p class="wp-block-paragraph">This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means &#8220;the process was forked&#8221;, not &#8220;the process can answer HTTP&#8221;. The pod lands in the Service&#8217;s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.</p>



<h3 class="wp-block-heading">The readiness probe that lies</h3>



<p class="wp-block-paragraph">Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that returns 200 from a static handler.</strong> If <code>/healthz</code> is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that checks downstream dependencies.</strong> This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks <em>every replica</em> unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers &#8220;can this pod serve traffic&#8221;, not &#8220;is the whole system healthy&#8221;. Check the dependency in your app&#8217;s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.</p>



<h3 class="wp-block-heading">Let the startup probe own the boot budget</h3>



<p class="wp-block-paragraph">For anything slow to boot, do not stretch <code>initialDelaySeconds</code> on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.</p>



<pre class="wp-block-code"><code># The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3</code></pre>



<p class="wp-block-paragraph">Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: traffic arrives after the pod has stopped</h2>



<p class="wp-block-paragraph">This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has &#8220;already added readiness probes&#8221;.</p>



<p class="wp-block-paragraph">When a pod is deleted, two things happen <em>in parallel</em>, not in sequence:</p>



<ol class="wp-block-list">
<li>The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container&#8217;s main process.</li>

<li>The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.</li>
</ol>



<p class="wp-block-paragraph">Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.</p>



<h3 class="wp-block-heading">The preStop sleep, and what it does not do</h3>



<p class="wp-block-paragraph">The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.</p>



<pre class="wp-block-code"><code>spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15</code></pre>



<p class="wp-block-paragraph">If you are on an older cluster without the native <code>sleep</code> handler, the equivalent is an exec hook, which does require a shell and a <code>sleep</code> binary in the image:</p>



<pre class="wp-block-code"><code>lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]</code></pre>



<p class="wp-block-paragraph">Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application&#8217;s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.</p>



<p class="wp-block-paragraph">There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.</p>



<h3 class="wp-block-heading">The grace period is a shared budget</h3>



<p class="wp-block-paragraph">This trips people up. The <code>terminationGracePeriodSeconds</code> countdown starts when the pod is marked Terminating, which is <em>before</em> the preStop hook runs, not after. The hook and your application&#8217;s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.</p>



<p class="wp-block-paragraph">Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.</p>



<p class="wp-block-paragraph">One more detail worth knowing before you go looking for it: probes accept their own <code>terminationGracePeriodSeconds</code> override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the rollout deletes capacity faster than it adds it</h2>



<p class="wp-block-paragraph">The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.</p>



<pre class="wp-block-code"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1</code></pre>



<p class="wp-block-paragraph">Setting <code>maxUnavailable: 0</code> is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise <code>maxSurge</code> to a percentage if you have the node capacity and want the speed back.</p>



<p class="wp-block-paragraph"><code>minReadySeconds</code> is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.</p>



<p class="wp-block-paragraph">Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.</p>



<pre class="wp-block-code"><code>apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api</code></pre>



<h2 class="wp-block-heading">Failure family four: the load balancer was never watching EndpointSlices</h2>



<p class="wp-block-paragraph">Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.</p>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.</p>



<p class="wp-block-paragraph">Two things help here:</p>



<ul class="wp-block-list">
<li><strong>Pod readiness gates.</strong> These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.</li>

<li><strong>Match your preStop sleep to the deregistration delay.</strong> If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.</li>
</ul>



<p class="wp-block-paragraph">The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod&#8217;s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.</p>



<h2 class="wp-block-heading">Verifying a zero-downtime rollout instead of hoping</h2>



<p class="wp-block-paragraph">A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.</p>



<ol class="wp-block-list">
<li><strong>Generate steady traffic against the real ingress path.</strong> Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.</li>

<li><strong>Trigger a rollout with no image change</strong>, so you are testing the mechanism rather than your new code.</li>

<li><strong>Watch EndpointSlices in a second terminal</strong> and note how long a terminating pod stays in the list.</li>

<li><strong>Count non-200 responses.</strong> Anything above zero is a bug, not noise.</li>

<li><strong>Repeat under realistic concurrency.</strong> A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.</li>
</ol>



<pre class="wp-block-code"><code># Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m</code></pre>



<p class="wp-block-paragraph">Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Rollout hangs, new pods never become Ready.</strong> The readiness probe is failing. <code>kubectl describe pod</code> shows the probe failure and the response it got. Check that the port name in the probe matches a declared <code>containerPort</code> name, that the path exists, and that the app is listening on all interfaces rather than only <code>127.0.0.1</code>, which is a classic one when moving from a local docker-compose setup.</p>



<p class="wp-block-paragraph"><strong>Probes fail only under load.</strong> Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within <code>timeoutSeconds</code>, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.</p>



<p class="wp-block-paragraph"><strong>Errors persist after adding a preStop sleep.</strong> Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.</p>



<p class="wp-block-paragraph"><strong>Pods stuck Terminating for the full grace period.</strong> Your app is ignoring SIGTERM. This is extremely common when the container&#8217;s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of <code>ENTRYPOINT</code>, or an init like <code>tini</code>, so your process actually receives the signal.</p>



<p class="wp-block-paragraph"><strong>Rollout succeeded but the new version is broken.</strong> Roll back first, investigate second.</p>



<pre class="wp-block-code"><code># Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.</li>

<li>Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.</li>

<li>Leaving <code>maxUnavailable</code> at the default on a service running near capacity.</li>

<li>Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.</li>

<li>Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.</li>

<li>Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.</li>

<li>Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Three separate endpoints: <code>/healthz/started</code>, <code>/healthz/ready</code>, <code>/healthz/live</code>. They answer different questions and should be allowed to disagree.</li>

<li><code>maxUnavailable: 0</code> plus a <code>maxSurge</code> you have node capacity for, as the default for anything user-facing.</li>

<li>A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.</li>

<li>A grace period that covers preStop plus your slowest realistic request, with slack.</li>

<li>Handle SIGTERM properly in the application, and make sure it reaches PID 1.</li>

<li>A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.</li>

<li><code>minReadySeconds</code> long enough to catch a pod that passes its probe and then dies.</li>

<li>Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do readiness probes alone give me zero-downtime deployments?</h3>



<p class="wp-block-paragraph">No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.</p>



<h3 class="wp-block-heading">How long should the preStop sleep be?</h3>



<p class="wp-block-paragraph">Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.</p>



<h3 class="wp-block-heading">Should the readiness probe check the database?</h3>



<p class="wp-block-paragraph">Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.</p>



<h3 class="wp-block-heading">What is the difference between maxSurge and maxUnavailable?</h3>



<p class="wp-block-paragraph"><code>maxSurge</code> is how many pods you may run <em>above</em> the replica count during a rollout. <code>maxUnavailable</code> is how many you may drop <em>below</em> it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.</p>



<h3 class="wp-block-heading">Why do I still get 502s on EKS, GKE or another managed cluster?</h3>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group&#8217;s deregistration delay rather than against kube-proxy.</p>



<h3 class="wp-block-heading">Can I get zero-downtime rollouts with a StatefulSet?</h3>



<p class="wp-block-paragraph">The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.</p>



<h3 class="wp-block-heading">Does kubectl rollout status prove the deploy was clean?</h3>



<p class="wp-block-paragraph">It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.</p>



<p class="wp-block-paragraph">Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your rollouts?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change</li>

<li>Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number</li>

<li>Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager</li>

<li>Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths</li>

<li>Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks</li>

<li>Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event</li>
</ul>



<p class="wp-block-paragraph">If you have a Deployment manifest, a <code>kubectl describe pod</code> output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
