<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Load Balancing | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/load-balancing/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/load-balancing/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 16:15:48 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Load Balancing | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/load-balancing/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>ALB vs API Gateway: Pick the Limits You Can Live With</title>
		<link>https://john-nessime.com/blog/devops/alb-vs-api-gateway/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[JWKS]]></category>
		<category><![CDATA[JWT Validation]]></category>
		<category><![CDATA[Listener Rules]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=550</guid>

					<description><![CDATA[<p>Choosing between an Application Load Balancer and API Gateway is not a feature comparison. It is a choice of hard limits, a billing curve and an authentication story. Here is how each one bills you, which constraints surface months later, what ALB's native JWT validation changes, and a decision procedure you can run in ten minutes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the reporting endpoint returns a 504 after about thirty seconds. The application log said the job finished in forty-one seconds and wrote a perfectly good response to a client that had already hung up. Nothing crashed, nothing scaled wrong, nobody deployed anything that week.</p>



<p class="wp-block-paragraph">The integration timeout had been sitting in front of that API since the day someone created it. It only became a bug when a customer&#8217;s dataset got big enough to cross it.</p>



<p class="wp-block-paragraph">That is the real shape of the ALB vs API Gateway decision. You&#8217;re not picking a feature list. You&#8217;re picking a set of hard limits, a billing curve and an authentication story, then living inside them. The failure mode that bites isn&#8217;t a bad Friday, it&#8217;s finding the constraint months later, once the thing in front of your app has become load-bearing.</p>



<p class="wp-block-paragraph">Below: how each one bills you, which limits surface late, what changed recently on the load balancer side, and a decision procedure.</p>



<h2 class="wp-block-heading">The old rule of thumb is shakier than it looks</h2>



<p class="wp-block-paragraph">The advice you find everywhere is: containers and instances behind an Application Load Balancer, serverless behind API Gateway. Wrong often enough to cost you. ALB takes Lambda functions as targets. API Gateway reaches containers through private integrations with an NLB, an ALB, or Cloud Map. Compute type barely narrows the field.</p>



<p class="wp-block-paragraph">What separates them is who&#8217;s calling and what needs enforcing before a request reaches your code. A browser session against your own web app is a different problem from a partner&#8217;s server calling a documented endpoint with a key you issued and a quota you promised. One needs routing and a session. The other needs a contract.</p>



<h2 class="wp-block-heading">Application Load Balancer: routing you own the position of</h2>



<p class="wp-block-paragraph">ALB is a Layer 7 router inside your VPC. You give it subnets and security groups, attach listeners, and write rules that match on host, path, header, query string, method or source IP, then forward, redirect, return a fixed response, or authenticate.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Long requests.</strong> The idle timeout is an attribute you set, defaulting to 60 seconds. Report generation and slow third-party calls become a number in Terraform instead of an architecture problem.</li>

<li><strong>Big bodies and streaming.</strong> Against instance, IP and container targets there&#8217;s no small payload ceiling to design around, and WebSocket upgrades pass through.</li>

<li><strong>Browser authentication.</strong> An HTTPS listener rule can run an <code>authenticate-oidc</code> or <code>authenticate-cognito</code> action before forwarding. The load balancer does the redirect dance, sets a session cookie, and hands your target the verified claims in a header. Your app reads a header instead of implementing OAuth.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>No per-client rate limiting.</strong> No API keys with quotas attached. Rate limiting means AWS WAF rate-based rules in front, counting requests per source over a window. Blunter than a per-consumer plan, and a separate bill.</li>

<li><strong>No request validation or response caching.</strong> A malformed body reaches your handler and burns your compute. A cacheable GET is recomputed unless you put a CDN in front.</li>

<li><strong>Lambda targets are constrained.</strong> Request body capped at 1 MB, response JSON capped at 1 MB, WebSocket upgrades rejected with a 400. If your function returns a large document, that ceiling arrives as a 502 in production, not an error at deploy time.</li>

<li><strong>You pay for it while it sleeps.</strong> An hourly charge per load balancer whether or not a request arrives. Three environments, three floors.</li>
</ul>



<h2 class="wp-block-heading">Amazon API Gateway: the contract layer</h2>



<p class="wp-block-paragraph">API Gateway isn&#8217;t a load balancer with extra features. It&#8217;s a place to define what your API promises: which routes exist, who may call them, how often, and in what shape. That framing explains most of its trade-offs.</p>



<p class="wp-block-paragraph">Choosing it is really two decisions, because REST APIs and HTTP APIs are different products under one name.</p>



<ul class="wp-block-list">
<li><strong>REST APIs</strong> carry the full management surface: API keys, usage plans, per-client throttling, request validation, response caching, AWS WAF integration, the private endpoint type, execution logs and X-Ray tracing.</li>

<li><strong>HTTP APIs</strong> are the stripped-down, cheaper option. They add a built-in JWT authorizer, automatic deployments and Cloud Map private integrations. They offer none of the six management features listed above.</li>
</ul>



<p class="wp-block-paragraph">That second line is the one people get wrong. HTTP APIs aren&#8217;t &#8220;REST APIs but cheaper&#8221;, they&#8217;re a different product that happens to serve HTTP. Pick them to save money, then discover you need metered access, and you&#8217;re rebuilding usage plans in your own code.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>You&#8217;re selling or governing access.</strong> API keys tied to usage plans, on REST APIs, are the only native answer to &#8220;customer B must not burn customer A&#8217;s capacity&#8221;.</li>

<li><strong>Rejecting bad requests before they cost you.</strong> Request validation against a model means a malformed payload never invokes your function. On per-invocation compute that&#8217;s a security boundary and a cost lever at once.</li>

<li><strong>Zero idle cost.</strong> No hourly floor, so a dev stage nobody calls is free. For genuinely spiky traffic this is often the whole argument, and there&#8217;s no network to design either.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>The integration timeout.</strong> The default is 29 seconds. It can be raised, but AWS notes that going beyond 29 seconds may require reducing your Region-level throttle quota. You&#8217;re buying request duration with request rate.</li>

<li><strong>Payload ceilings.</strong> There&#8217;s a documented maximum payload size, well below what a file upload service wants. Large objects want presigned S3 URLs either way, but here the gateway forces that conversation early.</li>

<li><strong>Cost scales with request count.</strong> Per request plus data transfer. A health check hammered by a monitoring tool shows up directly on the line item.</li>

<li><strong>The account throttle is shared.</strong> The default steady-state rate and burst quota applies per Region across your APIs in that account. Raisable, but a shared ceiling by default rather than a per-API one.</li>
</ul>



<h2 class="wp-block-heading">What changed: the load balancer validates JWTs itself now</h2>



<p class="wp-block-paragraph">For years the clean argument for API Gateway on machine-to-machine APIs was token validation. If a partner&#8217;s service calls you with a bearer token, someone has to check signature, issuer and expiry, and you&#8217;d rather that not be a library duplicated in every service.</p>



<p class="wp-block-paragraph">ALB does this natively now. A rule on an HTTPS listener can carry a <code>jwt-validation</code> action that runs before any routing action. The load balancer fetches signing keys from a JWKS endpoint you configure, checks the signature, and requires <code>iss</code> and <code>exp</code>. It also validates <code>nbf</code> and <code>iat</code> when present, and you can configure up to ten additional claims. Valid tokens forward unchanged. Invalid ones never reach your VPC.</p>



<p class="wp-block-paragraph">This isn&#8217;t <code>authenticate-oidc</code>, which is built for humans in browsers and works by redirecting. JWT validation does no redirect. It inspects the token and returns a yes or no, which is what service-to-service traffic needs. Validation action first, routing action second, and order matters:</p>



<pre class="wp-block-code"><code>aws elbv2 create-rule 
  --listener-arn "$LISTENER_ARN" 
  --priority 10 
  --conditions Field=path-pattern,Values="/api/*" 
  --actions '[
    {
      "Type": "jwt-validation",
      "JwtValidationConfig": {
        "JwksEndpoint": "https://issuer.example.com/.well-known/jwks.json",
        "Issuer": "https://issuer.example.com"
      },
      "Order": 1
    },
    {
      "Type": "forward",
      "TargetGroupArn": "$TARGET_GROUP_ARN",
      "Order": 2
    }
  ]'</code></pre>



<p class="wp-block-paragraph">Two constraints first. Only RS256 is supported, and the JWKS endpoint has a maximum response size and key count. A provider publishing a large key set fails that check, and the failure mode is that requests stop reaching your targets. Fetch your provider&#8217;s JWKS document and look at it before committing.</p>



<p class="wp-block-paragraph">So &#8220;we need token validation at the edge&#8221; is no longer by itself a reason to reach for API Gateway. What the gateway still owns is everything downstream of that yes or no: keys, quotas, per-consumer throttles, shape validation, caching.</p>



<h2 class="wp-block-heading">How each one actually bills you</h2>



<p class="wp-block-paragraph">This is where the argument usually goes wrong, because people compare unit prices instead of curve shapes.</p>



<p class="wp-block-paragraph">API Gateway is a slope with no floor: per request, plus data out. Zero traffic costs nothing, ten times the traffic costs roughly ten times as much.</p>



<p class="wp-block-paragraph">ALB is a floor plus a stranger slope. An hourly rate per load balancer, plus capacity units measured across four dimensions at once:</p>



<ul class="wp-block-list">
<li>New connections per second</li>

<li>Active connections per minute</li>

<li>Processed bytes per hour</li>

<li>Rule evaluations per second, where the first ten processed rules per request are free</li>
</ul>



<p class="wp-block-paragraph">You&#8217;re charged on the highest of the four in a given hour, not the sum. That mechanic explains most surprises. Clients that open a fresh connection per call can bill you on new connections while your bandwidth graph looks flat. A listener with dozens of path rules can bill you on rule evaluations while everything else idles.</p>



<p class="wp-block-paragraph">Two details worth knowing: Lambda targets get a smaller processed-bytes allowance per capacity unit than instance, container and IP targets, so identical traffic consumes more units. Enabling mutual TLS reduces the active-connections allowance per unit.</p>



<p class="wp-block-paragraph">Before modelling anything, look at what you already consume:</p>



<pre class="wp-block-code"><code>aws cloudwatch get-metric-statistics 
  --namespace AWS/ApplicationELB 
  --metric-name ConsumedLCUs 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --start-time "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" 
  --period 3600 
  --statistics Sum Maximum</code></pre>



<p class="wp-block-paragraph">Run that against a representative week and you have the real input. For API Gateway, the equivalent is request count per stage. Across several accounts, a cost visibility tool like Vantage or CloudZero attributes this per environment faster than Cost Explorer, though disciplined tagging gets you most of the way for free.</p>



<p class="wp-block-paragraph">Rough shape: low steady traffic favours API Gateway because you skip the floor. High-volume, byte-heavy, long-lived-connection traffic favours ALB because the per-request slope disappears. Chatty traffic with tiny payloads is where the curves cross, and that one is worth calculating.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for ALB vs API Gateway</h2>



<p class="wp-block-paragraph">Run these in order, stop at the first clear answer.</p>



<ol class="wp-block-list">
<li><strong>Does any request legitimately run past about thirty seconds?</strong> If yes and you can&#8217;t make it asynchronous, ALB. Raising the gateway timeout is possible but trades against your regional throttle quota.</li>

<li><strong>Do you need per-consumer quotas or metered access?</strong> If yes, API Gateway REST APIs. Nothing else here gives you keys and usage plans natively.</li>

<li><strong>Do requests or responses carry large payloads?</strong> If yes, ALB with instance or container targets. Both options constrain size, but the Lambda-target and gateway ceilings arrive sooner.</li>

<li><strong>Is traffic occasional, spiky, or mostly zero?</strong> If yes, API Gateway. An hourly floor for a few thousand requests a day is real cost with no matching benefit.</li>

<li><strong>Is the caller a browser session against your own app?</strong> If yes, ALB, with an authenticate action if you need login. The gateway&#8217;s management features are dead weight here.</li>

<li><strong>Is the caller another service with a bearer token and nothing more?</strong> Either works now. Decide on cost curve and payload shape, because JWT validation stopped being the tiebreaker.</li>
</ol>



<p class="wp-block-paragraph">Still torn? Choose the limits you&#8217;d rather explain to a customer in a year. Usually ALB for applications, API Gateway for products. Whichever you pick, alarm on the constraint rather than the symptom: integration latency approaching the timeout, or consumed capacity units trending up. Grafana Cloud or plain CloudWatch alarms both do the job. Then send a payload just over your ceiling in staging and confirm what the client actually receives.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;It&#8217;s serverless, so it&#8217;s cheaper&#8221;</h3>



<p class="wp-block-paragraph">Per-request billing is cheaper at low volume and more expensive at high volume. Arithmetic, not philosophy. The crossover depends on your request size and connection reuse, which is why the capacity-unit metric matters more than anyone&#8217;s price table.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll put API Gateway in front of it later&#8221;</h3>



<p class="wp-block-paragraph">You can, through a private integration, and it&#8217;s a legitimate pattern. But &#8220;later&#8221; means a second hop, a second bill, a second set of timeouts and a second place to look during an incident. Adding it deliberately is fine. Assuming it&#8217;s free is not.</p>



<h3 class="wp-block-heading">&#8220;ALB has no rate limiting, so it&#8217;s insecure&#8221;</h3>



<p class="wp-block-paragraph">ALB has no <em>native</em> rate limiting, but WAF rate-based rules in front of it are a real control, and a source-based limit is often closer to what you wanted than a per-key quota. Cloudflare in front of the load balancer is another common answer when you already want its caching and bot controls. What ALB genuinely lacks is the per-consumer contract, which matters when consumers are customers and doesn&#8217;t when they&#8217;re your own front end.</p>



<p class="wp-block-paragraph">Check the pairing before you rely on it, though: WAF associates with ALB and with API Gateway REST APIs, but not with HTTP APIs. And the amount of request body WAF reads is configurable for some resource types and fixed for the load balancer. Anything past that window isn&#8217;t scanned.</p>



<h3 class="wp-block-heading">&#8220;The front door has to be an AWS service&#8221;</h3>



<p class="wp-block-paragraph">For one service with predictable traffic, a plain VPS running Nginx or Caddy does the same job with far less surface. Providers like Contabo or InterServer are reasonable homes for a small internal API. It won&#8217;t scale into a multi-service platform and it moves patching and certificate renewal onto you, but &#8220;which AWS service&#8221; is sometimes the wrong question.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use API Gateway and ALB together?</h3>



<p class="wp-block-paragraph">Yes. API Gateway supports private integrations with an Application Load Balancer, so the gateway handles keys, quotas and validation while the load balancer routes inside the VPC. Sound pattern for a public product API in front of an existing internal platform. It also doubles the hops, the bills and the places a request can die, so use it when you need both jobs done.</p>



<h3 class="wp-block-heading">Which is cheaper, ALB or API Gateway?</h3>



<p class="wp-block-paragraph">Neither, universally. API Gateway has no idle cost and charges per request, so it wins at low or intermittent volume. ALB charges an hourly floor plus capacity units billed on the highest of four dimensions, so it wins once the per-request slope dominates. Pull your consumed capacity units and request counts and do the arithmetic on your own traffic.</p>



<h3 class="wp-block-heading">Can ALB do rate limiting?</h3>



<p class="wp-block-paragraph">Not natively. You get it by associating AWS WAF with the load balancer and using rate-based rules, which limit by source over a time window. If you need per-key quotas, that&#8217;s an API Gateway REST API usage plan, or logic you write yourself.</p>



<h3 class="wp-block-heading">Does ALB support JWT authentication now?</h3>



<p class="wp-block-paragraph">Yes, through a <code>jwt-validation</code> action on an HTTPS listener rule. It verifies the signature against a JWKS endpoint and checks issuer and expiry, plus optional additional claims. Note the RS256-only restriction and the JWKS size and key count limits before designing around it.</p>



<h3 class="wp-block-heading">Should I use HTTP APIs or REST APIs?</h3>



<p class="wp-block-paragraph">REST APIs if you need any of: API keys, usage plans, per-client throttling, request validation, response caching, WAF integration, or a private endpoint type. HTTP APIs when you need none of those and want the built-in JWT authorizer at a lower price. Migrating between them isn&#8217;t a config flag, so decide before you publish a URL.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">ALB vs API Gateway isn&#8217;t a question about serverless versus containers, and it stopped being a question about token validation. It&#8217;s a question about whether the thing in front of your application needs to enforce a contract with people outside your organisation.</p>



<p class="wp-block-paragraph">If it does, take the gateway and accept its timeout and payload ceilings as the price of keys, quotas and validation. If it doesn&#8217;t, take the load balancer and spend the saved effort on the network and the health checks. Either way, write down the three numbers that will eventually bite: timeout, payload ceiling, throttle quota. The 504 in that opening ticket wasn&#8217;t a bug. It was a decision nobody had written down.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your front door?</h2>



<p class="wp-block-paragraph">Cheap to get right at the start, expensive to revisit. If you&#8217;re staring at this now, or already living with the wrong answer:</p>



<ul class="wp-block-list">
<li>Modelling ALB capacity units against API Gateway request counts on your actual traffic, so the comparison uses your numbers</li>

<li>Auditing an existing gateway or load balancer for the limits you&#8217;re closest to hitting: timeouts, payload ceilings, throttle quotas</li>

<li>Moving browser authentication off your application onto an ALB authenticate rule, or setting up JWT validation at the listener for service-to-service traffic</li>

<li>Designing the gateway plus private integration pattern when you need both layers, including where to terminate TLS and how to preserve client IPs</li>

<li>Building the Terraform or OpenTofu modules for either option, with listener rules, target groups, health checks and alarms in one place</li>

<li>Setting up alarms that fire on approaching limits rather than the 5XX that follows</li>
</ul>



<p class="wp-block-paragraph">Send me a listener rule dump, a stage configuration, or a week of consumed capacity unit metrics and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</title>
		<link>https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Fargate]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Lightsail]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Managed Hosting]]></category>
		<category><![CDATA[NAT Gateway]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=262</guid>

					<description><![CDATA[<p>Choosing between Lightsail, EC2 and ECS for a small client workload is not a performance question. It is a question about which constraints you are accepting and how expensive they are to reverse. Profiles of all three with where each wins and loses, the cost levers that actually move the bill, a six-step decision procedure, and the arguments that fall apart on contact with a real project.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;site feels slow, can you take a look.&#8221; What it turned into was a migration. The client&#8217;s Lightsail instance had outgrown its bundle, the plan was to move to EC2, and Lightsail ships an export-to-EC2 feature that makes that sound like a button press.</p>



<p class="wp-block-paragraph">The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.</p>



<p class="wp-block-paragraph">That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.</p>



<p class="wp-block-paragraph">This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn&#8217;t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.</p>



<h2 class="wp-block-heading">Why the exit path matters more than the launch</h2>



<p class="wp-block-paragraph">All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.</p>



<p class="wp-block-paragraph">Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which is cheapest for a small site.&#8221; It is &#8220;how much of this stack is portable, and who operates it after handover.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Amazon Lightsail: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.</li>

<li>The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.</li>

<li>Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.</li>

<li>IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the <em>default</em> VPC in the same account and the same region. Not a custom VPC. Not another account.</li>

<li>Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.</li>

<li>The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.</li>

<li>No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.</li>
</ul>



<p class="wp-block-paragraph">Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail&#8217;s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.</p>



<h2 class="wp-block-heading">Amazon EC2: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.</li>

<li>Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.</li>

<li>Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.</li>

<li>Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.</li>

<li>Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.</li>

<li>Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.</li>

<li>You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.</li>
</ul>



<p class="wp-block-paragraph">The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.</p>



<h2 class="wp-block-heading">Amazon ECS: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.</p>



<ul class="wp-block-list">
<li><strong>ECS on Fargate</strong> bills per vCPU-hour and GB-hour of the resources you <em>declare in the task definition</em>, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.</li>

<li><strong>ECS on EC2</strong> means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.</li>
</ul>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.</li>

<li>Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.</li>

<li>Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.</li>

<li>Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.</li>

<li>If the app is already containerized and CI already builds images, you are most of the way there.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.</li>

<li>Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.</li>

<li>Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.</li>

<li>Handover is harder. A client&#8217;s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.</li>
</ul>



<p class="wp-block-paragraph">The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.</p>



<pre class="wp-block-code"><code># Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"</code></pre>



<p class="wp-block-paragraph">Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.</p>



<h2 class="wp-block-heading">Lightsail vs EC2 vs ECS on the lines that move the bill</h2>



<p class="wp-block-paragraph">Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.</p>



<ul class="wp-block-list">
<li><strong>Egress.</strong> Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.</li>

<li><strong>NAT gateway.</strong> Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.</li>

<li><strong>Load balancing.</strong> Lightsail&#8217;s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.</li>

<li><strong>Public IPv4.</strong> Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.</li>

<li><strong>Log ingestion.</strong> CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.</li>

<li><strong>Idle resources.</strong> Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.</li>
</ul>



<h2 class="wp-block-heading">A decision procedure for Lightsail vs EC2 vs ECS</h2>



<p class="wp-block-paragraph">Run these in order. The first one that gives a clear answer is the answer.</p>



<ol class="wp-block-list">
<li><strong>Does this workload need to talk privately to other AWS resources?</strong> If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.</li>

<li><strong>Who operates this in a year?</strong> If the answer is &#8220;the client&#8217;s office manager&#8221; or &#8220;nobody,&#8221; bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.</li>

<li><strong>Is the app already containerized, with CI that builds images?</strong> If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.</li>

<li><strong>Is the load steady or spiky?</strong> Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.</li>

<li><strong>How much egress?</strong> Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail&#8217;s bundled allowance is a real argument.</li>

<li><strong>What is the audit story?</strong> If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.</li>
</ol>



<p class="wp-block-paragraph">If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works <em>before</em> you need it. The commands are short:</p>



<pre class="wp-block-code"><code># Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records</code></pre>



<p class="wp-block-paragraph">Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.</p>



<p class="wp-block-paragraph">If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:</p>



<pre class="wp-block-code"><code>aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1</code></pre>



<p class="wp-block-paragraph">You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.</p>



<p class="wp-block-paragraph">Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;Lightsail is just EC2 with a nicer console&#8221;</h3>



<p class="wp-block-paragraph">Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.</p>



<h3 class="wp-block-heading">&#8220;Start on Lightsail, migrate to EC2 when you outgrow it&#8221;</h3>



<p class="wp-block-paragraph">Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.</p>



<h3 class="wp-block-heading">&#8220;Fargate is serverless, so it scales to zero and costs nothing when idle&#8221;</h3>



<p class="wp-block-paragraph">The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word &#8220;serverless&#8221; gives you.</p>



<h3 class="wp-block-heading">&#8220;Containers are better practice, so ECS is the professional choice&#8221;</h3>



<p class="wp-block-paragraph">Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.</p>



<h3 class="wp-block-heading">&#8220;Reserved Instances make EC2 cheaper, so always pick EC2&#8221;</h3>



<p class="wp-block-paragraph">Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client&#8217;s behalf is a conversation, not a default.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Lightsail cheaper than EC2 for a small website?</h3>



<p class="wp-block-paragraph">Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.</p>



<h3 class="wp-block-heading">Can a Lightsail instance connect to an RDS database?</h3>



<p class="wp-block-paragraph">Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the <em>default</em> VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.</p>



<h3 class="wp-block-heading">Does everything migrate when I export Lightsail to EC2?</h3>



<p class="wp-block-paragraph">No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel &amp; WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.</p>



<h3 class="wp-block-heading">Is ECS overkill for a single small container?</h3>



<p class="wp-block-paragraph">Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.</p>



<h3 class="wp-block-heading">Why is my Fargate bill higher than the vCPU-hour rate suggests?</h3>



<p class="wp-block-paragraph">Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.</p>



<h3 class="wp-block-heading">Should I use Lightsail container service instead of ECS?</h3>



<p class="wp-block-paragraph">Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.</p>



<p class="wp-block-paragraph">One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.</p>



<p class="wp-block-paragraph">Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion before you commit?</h2>



<p class="wp-block-paragraph">Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:</p>



<ul class="wp-block-list">
<li>Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses</li>

<li>Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency</li>

<li>Planning and executing Lightsail to EC2 migrations, including the parts that do not export</li>

<li>Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it</li>

<li>Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy</li>

<li>Writing the handover runbook so the next person can operate it without calling you</li>
</ul>



<p class="wp-block-paragraph">If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</title>
		<link>https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Hosting]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Automatic HTTPS]]></category>
		<category><![CDATA[Caddy]]></category>
		<category><![CDATA[Caddyfile]]></category>
		<category><![CDATA[Certbot]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Let's Encrypt]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[mod_php]]></category>
		<category><![CDATA[MPM]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reverse Proxy]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[TLS]]></category>
		<category><![CDATA[VPS]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[Web Server]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=195</guid>

					<description><![CDATA[<p>Benchmarks pick the wrong winner. Here is how Nginx, Apache and Caddy actually differ in production: how each one fails under load, who owns certificate renewal, and where configuration lives when a developer needs to change it. Includes a decision procedure and the mistakes I see most.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A message landed in a channel I was in: &#8220;We&#8217;re standardising on Nginx.&#8221; No traffic profile. No mention of what the current setup was actually failing at. No list of the applications that would have to move. The decision got made because everybody knows Nginx is fast.</p>



<p class="wp-block-paragraph">That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.</p>



<p class="wp-block-paragraph">So this post compares <strong>Nginx vs Apache vs Caddy</strong> on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.</p>



<h2 class="wp-block-heading">Why the benchmark comparison picks the wrong winner</h2>



<p class="wp-block-paragraph">Almost every throughput comparison you&#8217;ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you&#8217;re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.</p>



<p class="wp-block-paragraph">The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn&#8217;t most projects.</p>



<p class="wp-block-paragraph">For everyone else, the useful question is not &#8220;which is fastest&#8221; but &#8220;which chores am I choosing to keep.&#8221; That&#8217;s what the rest of this is about.</p>



<h2 class="wp-block-heading">Failure family one: the process model decides how you fall over</h2>



<p class="wp-block-paragraph">This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.</p>



<h3 class="wp-block-heading">Apache: a hard worker ceiling, and a queue behind it</h3>



<p class="wp-block-paragraph">Apache&#8217;s behaviour depends entirely on which Multi-Processing Module is loaded. With <code>prefork</code>, you get one process per connection, capped by <code>MaxRequestWorkers</code>. When you hit that cap, new connections don&#8217;t get rejected. They sit in the kernel&#8217;s listen backlog waiting for a worker to free up.</p>



<p class="wp-block-paragraph">The symptom is the worst kind: the site &#8220;hangs.&#8221; Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.</p>



<p class="wp-block-paragraph">Check which MPM you&#8217;re actually running before you theorise about anything:</p>



<pre class="wp-block-code"><code># RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"</code></pre>



<p class="wp-block-paragraph">The reason prefork survives is <code>mod_php</code>. It isn&#8217;t safe with Apache&#8217;s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over <code>mod_proxy_fcgi</code> instead, which frees you to run <code>mpm_event</code> and gets you HTTP/2 as a side effect, since <code>mod_http2</code> disables itself under prefork.</p>



<p class="wp-block-paragraph">If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.</p>



<h3 class="wp-block-heading">Nginx: cheap connections, different limits</h3>



<p class="wp-block-paragraph">Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.</p>



<p class="wp-block-paragraph">Its ceiling is <code>worker_processes</code> multiplied by <code>worker_connections</code>, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That&#8217;s a genuine operational advantage: the failure announces itself.</p>



<p class="wp-block-paragraph">The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.</p>



<h3 class="wp-block-heading">Caddy: goroutines, and a softer edge</h3>



<p class="wp-block-paragraph">Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don&#8217;t tune a worker count at all.</p>



<p class="wp-block-paragraph">The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean &#8220;workers exhausted&#8221; line, you get memory growth and garbage collection pressure. There&#8217;s no knob you turned wrong, which is pleasant right up until you need to explain what happened.</p>



<h2 class="wp-block-heading">Failure family two: who owns the certificate lifecycle</h2>



<p class="wp-block-paragraph">This is the one that actually causes outages, and it is the sharpest genuine difference between the three.</p>



<p class="wp-block-paragraph">With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.</p>



<p class="wp-block-paragraph">The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:</p>



<pre class="wp-block-code"><code># Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"</code></pre>



<p class="wp-block-paragraph">The distinction that matters: <code>--deploy-hook</code> runs only when a certificate was actually renewed, so you&#8217;re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.</p>



<p class="wp-block-paragraph">Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.</p>



<p class="wp-block-paragraph">It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy&#8217;s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an <code>ask</code> endpoint you host:</p>



<pre class="wp-block-code"><code>{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}</code></pre>



<p class="wp-block-paragraph">Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older <code>interval</code> and <code>burst</code> rate-limiting options are no longer recommended and should be removed if you still have them in a config.</p>



<p class="wp-block-paragraph">One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.</p>



<h2 class="wp-block-heading">Failure family three: where the configuration lives</h2>



<p class="wp-block-paragraph">This is the axis people underweight, and it&#8217;s usually the one that actually decides the answer.</p>



<p class="wp-block-paragraph">Apache supports <code>.htaccess</code>: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.</p>



<p class="wp-block-paragraph">The costs are real and they&#8217;re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for <code>.htaccess</code> files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.</p>



<p class="wp-block-paragraph">Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with <code>try_files</code> rather than per-directory rule files. If you own the box, that&#8217;s a feature: nothing changes routing behind your back and the running configuration is exactly what&#8217;s in version control. If you host other people&#8217;s applications, it&#8217;s a support burden, because every customer rewrite rule becomes a ticket for you.</p>



<p class="wp-block-paragraph">Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Nginx: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you&#8217;ll meet has a known-good Nginx config, and everyone you hire has already read one.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the DNS behaviour catches people constantly. A hostname written directly into <code>proxy_pass</code>, or into an <code>upstream</code> block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says &#8220;stale DNS&#8221;; you just get connection timeouts to an address that looks plausible.</p>



<p class="wp-block-paragraph">Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that&#8217;s what moves the lookup from config parse time to request time:</p>



<pre class="wp-block-code"><code>location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}</code></pre>



<p class="wp-block-paragraph">Note the <code>$request_uri</code> on the end. Once a variable appears in <code>proxy_pass</code>, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream&#8217;s root. Also worth knowing: an <code>upstream</code> block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.</p>



<p class="wp-block-paragraph">The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But &#8220;who decides what ships&#8221; is a fair question to ask about any dependency, and this one now has a public answer.</p>



<h2 class="wp-block-heading">Apache: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> memory per connection, if you&#8217;re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that&#8217;s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.</p>



<p class="wp-block-paragraph">I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and <code>mod_php</code> to <code>mpm_event</code> and PHP-FPM first, then measure, and quite often stop there.</p>



<h2 class="wp-block-heading">Caddy: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.</p>



<pre class="wp-block-code"><code>app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}</code></pre>



<p class="wp-block-paragraph">That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with <code>xcaddy</code> rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there&#8217;s a subtler cost: automation you don&#8217;t understand is still automation you&#8217;re responsible for. Being paged about ACME behaviour you&#8217;ve never had to learn is a worse afternoon than being paged about Certbot.</p>



<p class="wp-block-paragraph">On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the <code>dynamic a</code> upstream module with a <code>refresh</code> interval is the built-in answer, which is a cleaner story than Nginx&#8217;s variable trick.</p>



<h2 class="wp-block-heading">A decision procedure for Nginx vs Apache vs Caddy</h2>



<p class="wp-block-paragraph">Work down this list and stop at the first one that matches. It&#8217;ll be right more often than a benchmark.</p>



<ol class="wp-block-list">
<li><strong>Do applications you don&#8217;t control need to change server behaviour?</strong> If yes, Apache. Nothing else has <code>.htaccess</code>, and reimplementing it in tickets is not a plan.</li>

<li><strong>Is TLS renewal your top operational risk?</strong> Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.</li>

<li><strong>Are you terminating a very high connection rate, or serving heavy static content?</strong> Nginx. This is the workload it was designed around.</li>

<li><strong>Does the team already run one of these well?</strong> Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.</li>

<li><strong>Is there a working Apache install and no specific complaint?</strong> Fix the MPM and PHP handler, then reassess. Most &#8220;we need Nginx&#8221; tickets are actually &#8220;we need to stop using prefork.&#8221;</li>

<li><strong>None of the above?</strong> Nginx for anything that will be handed to another engineer. Caddy for anything you&#8217;ll run yourself.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Renewing certificates without reloading.</strong> The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.</li>

<li><strong>Migrating for benchmark numbers.</strong> You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.</li>

<li><strong>Leaving Apache on prefork after moving to PHP-FPM.</strong> The migration is only half done, and you&#8217;re still paying prefork&#8217;s memory cost while also giving up HTTP/2.</li>

<li><strong>Hostnames baked into <code>proxy_pass</code> with dynamic upstreams.</strong> Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.</li>

<li><strong>Enabling Caddy&#8217;s on-demand TLS without an <code>ask</code> endpoint.</strong> You&#8217;ve handed strangers the ability to make you request certificates.</li>

<li><strong>Exposing Caddy&#8217;s admin API beyond loopback.</strong> Unauthenticated full config replacement is exactly as bad as it sounds.</li>

<li><strong>Monitoring the certificate file instead of the handshake.</strong> The file is almost never the thing that&#8217;s wrong. Check what the server actually presents, from outside the box.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide, in practice</h2>



<ul class="wp-block-list">
<li>Pick based on who has to change the config six months from now, not on requests per second today.</li>

<li>Whichever you choose, dump the effective configuration rather than reading files: <code>nginx -T</code>, <code>apachectl -S</code>, <code>caddy adapt --config /etc/caddy/Caddyfile</code>. Include files hide surprises.</li>

<li>Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.</li>

<li>Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.</li>

<li>Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.</li>

<li>Validate before reloading, every time. <code>nginx -t</code>, <code>apachectl configtest</code> and <code>caddy validate</code> all exist so you don&#8217;t find out from users.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Caddy production ready?</h3>



<p class="wp-block-paragraph">Yes. It&#8217;s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.</p>



<h3 class="wp-block-heading">Can I run Nginx and Apache together?</h3>



<p class="wp-block-paragraph">Yes, and it&#8217;s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their <code>.htaccess</code> behaviour. You get Nginx&#8217;s connection handling and Apache&#8217;s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.</p>



<h3 class="wp-block-heading">Which is best for WordPress?</h3>



<p class="wp-block-paragraph">Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into <code>try_files</code> and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.</p>



<h3 class="wp-block-heading">Does Nginx really re-resolve DNS for upstreams?</h3>



<p class="wp-block-paragraph">Not by default. A literal hostname in <code>proxy_pass</code> or an <code>upstream</code> block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a <code>resolver</code> directive, and remembering that this changes how the request URI is passed.</p>



<h3 class="wp-block-heading">Should I switch from mod_php to PHP-FPM?</h3>



<p class="wp-block-paragraph">In almost all cases yes. It&#8217;s the change that unlocks Apache&#8217;s threaded MPMs and HTTP/2, and it separates PHP&#8217;s lifecycle from the web server&#8217;s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or <code>.htaccess</code>, since those move to a <code>.user.ini</code> file with different syntax.</p>



<h3 class="wp-block-heading">Does the freenginx fork mean I should stop using Nginx?</h3>



<p class="wp-block-paragraph">No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn&#8217;t put a client on a fork without a specific reason.</p>



<h3 class="wp-block-heading">Which uses the least memory?</h3>



<p class="wp-block-paragraph">Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on <code>mpm_event</code>, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Nginx vs Apache vs Caddy</strong> question is not a performance question. All three are fast enough for what you&#8217;re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.</p>



<p class="wp-block-paragraph">Pick the set of chores your team can actually carry. Then go fix the thing that&#8217;s genuinely slow, which is almost certainly not the web server.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your web server setup?</h2>



<p class="wp-block-paragraph">Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does</li>

<li>Migrating Apache from prefork and <code>mod_php</code> to <code>mpm_event</code> with PHP-FPM, without breaking existing <code>.htaccess</code> behaviour</li>

<li>Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches</li>

<li>Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring</li>

<li>Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted</li>

<li>Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me your config, an error log excerpt, or the output of <code>nginx -T</code> and tell me what you&#8217;re seeing. I&#8217;d rather look at the real thing than guess from a description.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/nginx-vs-apache-vs-caddy/">Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</title>
		<link>https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 11 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Health Checks]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=189</guid>

					<description><![CDATA[<p>Readiness probes are necessary for Kubernetes zero-downtime rollouts, but they only close one of four gaps. Here is what actually drops requests during a deploy: the race between SIGTERM and endpoint propagation, probes that lie, surge settings that quietly delete capacity, and load balancers that never watched EndpointSlices in the first place.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The rollout goes green. <code>kubectl rollout status</code> exits zero, every pod reports <code>1/1 Running</code>, and the deploy channel gets the thumbs-up emoji. Then someone from support pastes a screenshot: a spike of 502s, about forty seconds wide, lined up exactly with the deploy. Nobody paged. Nothing restarted. The graph just has a small cliff in it.</p>



<p class="wp-block-paragraph">That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that <strong>Kubernetes zero-downtime rollouts</strong> depend on, and it is not the question that usually bites.</p>



<p class="wp-block-paragraph">This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.</p>



<h2 class="wp-block-heading">What a zero-downtime rollout actually requires</h2>



<p class="wp-block-paragraph">Three independent things have to be true at the same time, and each is owned by a different part of the system.</p>



<ul class="wp-block-list">
<li><strong>Capacity never dips below what traffic needs.</strong> Owned by the Deployment&#8217;s rolling update strategy.</li>

<li><strong>No pod receives traffic before it can serve it.</strong> Owned by the readiness probe.</li>

<li><strong>No pod receives traffic after it stops serving it.</strong> Owned by graceful termination, and this is the one nobody configures.</li>
</ul>



<p class="wp-block-paragraph">Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.</p>



<h2 class="wp-block-heading">Failure family one: traffic arrives before the pod can serve it</h2>



<p class="wp-block-paragraph">This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means &#8220;the process was forked&#8221;, not &#8220;the process can answer HTTP&#8221;. The pod lands in the Service&#8217;s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.</p>



<h3 class="wp-block-heading">The readiness probe that lies</h3>



<p class="wp-block-paragraph">Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that returns 200 from a static handler.</strong> If <code>/healthz</code> is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.</p>



<p class="wp-block-paragraph"><strong>A readiness endpoint that checks downstream dependencies.</strong> This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks <em>every replica</em> unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers &#8220;can this pod serve traffic&#8221;, not &#8220;is the whole system healthy&#8221;. Check the dependency in your app&#8217;s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.</p>



<h3 class="wp-block-heading">Let the startup probe own the boot budget</h3>



<p class="wp-block-paragraph">For anything slow to boot, do not stretch <code>initialDelaySeconds</code> on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.</p>



<pre class="wp-block-code"><code># The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3</code></pre>



<p class="wp-block-paragraph">Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: traffic arrives after the pod has stopped</h2>



<p class="wp-block-paragraph">This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has &#8220;already added readiness probes&#8221;.</p>



<p class="wp-block-paragraph">When a pod is deleted, two things happen <em>in parallel</em>, not in sequence:</p>



<ol class="wp-block-list">
<li>The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container&#8217;s main process.</li>

<li>The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.</li>
</ol>



<p class="wp-block-paragraph">Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.</p>



<h3 class="wp-block-heading">The preStop sleep, and what it does not do</h3>



<p class="wp-block-paragraph">The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.</p>



<pre class="wp-block-code"><code>spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15</code></pre>



<p class="wp-block-paragraph">If you are on an older cluster without the native <code>sleep</code> handler, the equivalent is an exec hook, which does require a shell and a <code>sleep</code> binary in the image:</p>



<pre class="wp-block-code"><code>lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]</code></pre>



<p class="wp-block-paragraph">Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application&#8217;s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.</p>



<p class="wp-block-paragraph">There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.</p>



<h3 class="wp-block-heading">The grace period is a shared budget</h3>



<p class="wp-block-paragraph">This trips people up. The <code>terminationGracePeriodSeconds</code> countdown starts when the pod is marked Terminating, which is <em>before</em> the preStop hook runs, not after. The hook and your application&#8217;s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.</p>



<p class="wp-block-paragraph">Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.</p>



<p class="wp-block-paragraph">One more detail worth knowing before you go looking for it: probes accept their own <code>terminationGracePeriodSeconds</code> override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the rollout deletes capacity faster than it adds it</h2>



<p class="wp-block-paragraph">The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.</p>



<pre class="wp-block-code"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1</code></pre>



<p class="wp-block-paragraph">Setting <code>maxUnavailable: 0</code> is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise <code>maxSurge</code> to a percentage if you have the node capacity and want the speed back.</p>



<p class="wp-block-paragraph"><code>minReadySeconds</code> is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.</p>



<p class="wp-block-paragraph">Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.</p>



<pre class="wp-block-code"><code>apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api</code></pre>



<h2 class="wp-block-heading">Failure family four: the load balancer was never watching EndpointSlices</h2>



<p class="wp-block-paragraph">Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.</p>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.</p>



<p class="wp-block-paragraph">Two things help here:</p>



<ul class="wp-block-list">
<li><strong>Pod readiness gates.</strong> These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.</li>

<li><strong>Match your preStop sleep to the deregistration delay.</strong> If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.</li>
</ul>



<p class="wp-block-paragraph">The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod&#8217;s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.</p>



<h2 class="wp-block-heading">Verifying a zero-downtime rollout instead of hoping</h2>



<p class="wp-block-paragraph">A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.</p>



<ol class="wp-block-list">
<li><strong>Generate steady traffic against the real ingress path.</strong> Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.</li>

<li><strong>Trigger a rollout with no image change</strong>, so you are testing the mechanism rather than your new code.</li>

<li><strong>Watch EndpointSlices in a second terminal</strong> and note how long a terminating pod stays in the list.</li>

<li><strong>Count non-200 responses.</strong> Anything above zero is a bug, not noise.</li>

<li><strong>Repeat under realistic concurrency.</strong> A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.</li>
</ol>



<pre class="wp-block-code"><code># Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m</code></pre>



<p class="wp-block-paragraph">Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Rollout hangs, new pods never become Ready.</strong> The readiness probe is failing. <code>kubectl describe pod</code> shows the probe failure and the response it got. Check that the port name in the probe matches a declared <code>containerPort</code> name, that the path exists, and that the app is listening on all interfaces rather than only <code>127.0.0.1</code>, which is a classic one when moving from a local docker-compose setup.</p>



<p class="wp-block-paragraph"><strong>Probes fail only under load.</strong> Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within <code>timeoutSeconds</code>, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.</p>



<p class="wp-block-paragraph"><strong>Errors persist after adding a preStop sleep.</strong> Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.</p>



<p class="wp-block-paragraph"><strong>Pods stuck Terminating for the full grace period.</strong> Your app is ignoring SIGTERM. This is extremely common when the container&#8217;s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of <code>ENTRYPOINT</code>, or an init like <code>tini</code>, so your process actually receives the signal.</p>



<p class="wp-block-paragraph"><strong>Rollout succeeded but the new version is broken.</strong> Roll back first, investigate second.</p>



<pre class="wp-block-code"><code># Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.</li>

<li>Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.</li>

<li>Leaving <code>maxUnavailable</code> at the default on a service running near capacity.</li>

<li>Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.</li>

<li>Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.</li>

<li>Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.</li>

<li>Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Three separate endpoints: <code>/healthz/started</code>, <code>/healthz/ready</code>, <code>/healthz/live</code>. They answer different questions and should be allowed to disagree.</li>

<li><code>maxUnavailable: 0</code> plus a <code>maxSurge</code> you have node capacity for, as the default for anything user-facing.</li>

<li>A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.</li>

<li>A grace period that covers preStop plus your slowest realistic request, with slack.</li>

<li>Handle SIGTERM properly in the application, and make sure it reaches PID 1.</li>

<li>A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.</li>

<li><code>minReadySeconds</code> long enough to catch a pod that passes its probe and then dies.</li>

<li>Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do readiness probes alone give me zero-downtime deployments?</h3>



<p class="wp-block-paragraph">No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.</p>



<h3 class="wp-block-heading">How long should the preStop sleep be?</h3>



<p class="wp-block-paragraph">Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.</p>



<h3 class="wp-block-heading">Should the readiness probe check the database?</h3>



<p class="wp-block-paragraph">Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.</p>



<h3 class="wp-block-heading">What is the difference between maxSurge and maxUnavailable?</h3>



<p class="wp-block-paragraph"><code>maxSurge</code> is how many pods you may run <em>above</em> the replica count during a rollout. <code>maxUnavailable</code> is how many you may drop <em>below</em> it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.</p>



<h3 class="wp-block-heading">Why do I still get 502s on EKS, GKE or another managed cluster?</h3>



<p class="wp-block-paragraph">Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group&#8217;s deregistration delay rather than against kube-proxy.</p>



<h3 class="wp-block-heading">Can I get zero-downtime rollouts with a StatefulSet?</h3>



<p class="wp-block-paragraph">The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.</p>



<h3 class="wp-block-heading">Does kubectl rollout status prove the deploy was clean?</h3>



<p class="wp-block-paragraph">It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.</p>



<p class="wp-block-paragraph">Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your rollouts?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change</li>

<li>Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number</li>

<li>Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager</li>

<li>Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths</li>

<li>Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks</li>

<li>Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event</li>
</ul>



<p class="wp-block-paragraph">If you have a Deployment manifest, a <code>kubectl describe pod</code> output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/kubernetes-zero-downtime-rollouts/">Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn&#8217;t Saving You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
