<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>CloudWatch | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/cloudwatch/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/cloudwatch/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 02 Nov 2026 07:03:28 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>CloudWatch | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/cloudwatch/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</title>
		<link>https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 02 Nov 2026 07:03:28 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS CLI]]></category>
		<category><![CDATA[AWS WAF]]></category>
		<category><![CDATA[Bot Mitigation]]></category>
		<category><![CDATA[Brute Force Protection]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[DDoS Protection]]></category>
		<category><![CDATA[HTTP 429]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Throttling]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<category><![CDATA[X-Forwarded-For]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1023</guid>

					<description><![CDATA[<p>Your rate limit worked, and it also blocked your biggest customer's office NAT. How to pick aggregation keys, scope rules per endpoint, return 429 instead of 403, and roll out AWS WAF rate-based rules in Count mode first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/">AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like this: <em>&#8220;Your API started returning 403 to our integration around nine this morning. Nothing changed on our side.&#8221;</em> You check the application logs and there&#8217;s nothing there, because the requests never reached the application. They died at the edge, in a WAF rule somebody added last month to stop a credential-stuffing run on the login page.</p>



<p class="wp-block-paragraph">That rule worked. It also counted every request from the customer&#8217;s office NAT, their ERP connector and their warehouse scanners as one source, and at the start of the working day that one source crossed the line. The customer who sends you the most traffic is, almost by definition, the one most likely to look like an attacker to a counter that only sees IP addresses.</p>



<p class="wp-block-paragraph">This post is about building <strong>AWS WAF rate-based rules</strong> that catch floods, scrapers and brute-force attempts without throttling the people you actually want. I&#8217;ll go through the four ways these rules go wrong (the key, the scope, the response and the timing), a rollout order that keeps you out of trouble, and the troubleshooting steps for when a real user gets caught anyway.</p>



<h2 class="wp-block-heading">How AWS WAF rate-based rules actually count</h2>



<p class="wp-block-paragraph">A rate-based rule does three things. It decides which requests it cares about (the scope-down statement), it groups those requests into buckets (the aggregation key), and it counts each bucket over a sliding look-back window. When a bucket goes over your limit, the rule action applies to requests in that bucket until the rate falls back under.</p>



<p class="wp-block-paragraph">By default the bucket is the source IP address, and the window is five minutes. You can change both. AWS lets you pick a one, two, five or ten minute evaluation window, and you can aggregate on things other than IP: a forwarded IP header, a header value, a cookie, a query argument, the URI path, the HTTP method, an ASN, a JA3 or JA4 TLS fingerprint, or a label namespace set by an earlier rule. You can combine several of these into a composite key.</p>



<p class="wp-block-paragraph">Most bad outcomes come from one of those choices being made with the attacker&#8217;s traffic in mind and never checked against the customer&#8217;s.</p>



<h2 class="wp-block-heading">Failure family 1: the key groups the wrong people together</h2>



<h3 class="wp-block-heading">Shared egress IPs look like one very busy client</h3>



<p class="wp-block-paragraph">IP aggregation assumes one IP is roughly one client. That stopped being true a long time ago. A company office sits behind one NAT address. Mobile carriers put thousands of subscribers behind carrier-grade NAT. People browsing through a commercial VPN such as NordVPN or Surfshark share exit nodes with strangers. A B2B customer&#8217;s integration server, their staff browsers and their scheduled exports can all arrive from the same address.</p>



<p class="wp-block-paragraph">None of that is abuse, and it all lands in one bucket. A limit sized for &#8220;one heavy user&#8221; is really a limit on everyone behind that address, combined.</p>



<h3 class="wp-block-heading">The proxy in front of WAF hides the client entirely</h3>



<p class="wp-block-paragraph">This one is invisible until it isn&#8217;t. If your web ACL sits on an Application Load Balancer and CloudFront (or another CDN like Cloudflare) sits in front of that load balancer, the source IP that WAF sees on the ALB is the CDN&#8217;s edge address, not the visitor&#8217;s. A plain IP-aggregated rule is now counting &#8220;requests from this CloudFront edge location&#8221;. Under normal load that stays under the limit, so testing looks fine. Under a traffic spike, legitimate users sharing a busy edge get blocked together.</p>



<p class="wp-block-paragraph">There are two fixes, and they aren&#8217;t equal. The cleaner one is to attach the rate-based rule to a web ACL on the CloudFront distribution itself, where the source IP is the viewer. The other is forwarded IP aggregation on the ALB, which reads the client address from a header such as <code>X-Forwarded-For</code>. That brings its own problem.</p>



<h3 class="wp-block-heading">Forwarded IP aggregation trusts the part the client wrote</h3>



<p class="wp-block-paragraph">For rate-based rules, AWS WAF uses the <strong>first</strong> address in the forwarded header. Proxies append to <code>X-Forwarded-For</code>, so the first entry is whatever the client sent before any proxy touched the request. A client that sets its own <code>X-Forwarded-For</code> value, and rotates it on every request, lands in a new bucket every time. Your rate limit never fires for that client, while it keeps working perfectly for honest users who don&#8217;t forge headers.</p>



<p class="wp-block-paragraph">Two related details from the forwarded IP documentation matter here. If the header is missing, the rule doesn&#8217;t apply to that request at all, not even the fallback behavior. And the fallback (match or no match) only kicks in when the header is present but holds a malformed address. So forwarded IP aggregation is safe only when something upstream that you control overwrites the header rather than appending to it. If you can&#8217;t guarantee that, move the rule to the edge.</p>



<h3 class="wp-block-heading">Composite keys exclude requests that lack a component</h3>



<p class="wp-block-paragraph">Custom keys are how you stop punishing shared IPs. Aggregate on an API key header, a session cookie, or IP plus something else, and each customer gets their own bucket. The catch is in one line of the docs: every component in the key must be present, or the request isn&#8217;t evaluated by the rule at all.</p>



<p class="wp-block-paragraph">So a rule keyed on <code>x-api-key</code> rate-limits authenticated clients precisely and ignores anonymous traffic completely. The attacker who simply leaves the header off is invisible to it. You always need a second rule for the requests the first one can&#8217;t see, usually an IP-keyed rule scoped to requests without the header.</p>



<p class="wp-block-paragraph">Here&#8217;s the header-keyed rule. The limit is a placeholder; you&#8217;ll derive a real one from your own logs in a moment.</p>



<pre class="wp-block-code"><code>{
  "Name": "rate-per-api-key",
  "Priority": 20,
  "Statement": {
    "RateBasedStatement": {
      "Limit": 2000,
      "EvaluationWindowSec": 300,
      "AggregateKeyType": "CUSTOM_KEYS",
      "CustomKeys": [
        {
          "Header": {
            "Name": "x-api-key",
            "TextTransformations": [
              {
                "Priority": 0,
                "Type": "NONE"
              }
            ]
          }
        }
      ]
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "rate-per-api-key"
  }
}</code></pre>



<p class="wp-block-paragraph">Note the action is <code>Count</code>, not <code>Block</code>. More on why below. Also remember that a header value is only as trustworthy as your API validates it to be. A made-up key still gets its own bucket in WAF, so the backend has to reject it.</p>



<h2 class="wp-block-heading">Failure family 2: one limit for traffic with very different shapes</h2>



<p class="wp-block-paragraph">A single site-wide rule applies one number to everything. But a page load might fetch dozens of assets, an SPA can fire a burst of API calls on first render, and a login form should see a handful of posts per person per session. Any limit loose enough for the first case is useless for the last one.</p>



<p class="wp-block-paragraph">The answer is several narrow rules instead of one wide one, each with a scope-down statement:</p>



<ul class="wp-block-list">

<li><strong>Login, signup, password reset and OTP endpoints:</strong> tight limit, IP or forwarded IP key. This is where credential stuffing and SMS pumping live.</li>


<li><strong>Search, pricing and other expensive endpoints:</strong> moderate limit. Scrapers go here, and each request costs you real backend work.</li>


<li><strong>Authenticated API:</strong> key on the credential, sized from actual per-customer usage.</li>


<li><strong>Everything else:</strong> one generous catch-all on IP, meant for floods, not for shaping.</li>

</ul>



<p class="wp-block-paragraph">Static assets are often best left out of rate counting entirely, especially when a CDN is serving them from cache and they never reach anything expensive. Counting them mainly inflates the numbers for legitimate browsers.</p>



<p class="wp-block-paragraph">A scoped login rule looks like this:</p>



<pre class="wp-block-code"><code>{
  "Name": "rate-login",
  "Priority": 10,
  "Statement": {
    "RateBasedStatement": {
      "Limit": 100,
      "EvaluationWindowSec": 300,
      "AggregateKeyType": "IP",
      "ScopeDownStatement": {
        "ByteMatchStatement": {
          "SearchString": "/login",
          "FieldToMatch": {
            "UriPath": {}
          },
          "TextTransformations": [
            {
              "Priority": 0,
              "Type": "LOWERCASE"
            }
          ],
          "PositionalConstraint": "STARTS_WITH"
        }
      }
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "rate-login"
  }
}</code></pre>



<p class="wp-block-paragraph">The <code>LOWERCASE</code> transformation matters more than it looks. Without it, <code>/Login</code> may route to the same handler in your app but fall outside the scope-down match.</p>



<h3 class="wp-block-heading">Exempting a partner: Allow is heavier than it looks</h3>



<p class="wp-block-paragraph">When a known partner keeps tripping a limit, the tempting fix is an IP set rule with an <code>Allow</code> action placed above the rate rule. It works, but <code>Allow</code> is terminating. That partner&#8217;s traffic now skips every rule below it, including your managed rule groups for injection and known bad inputs. If their integration server is ever compromised, you&#8217;ve pre-approved the attacker.</p>



<p class="wp-block-paragraph">The narrower fix is to exclude the IP set inside the rate rule&#8217;s own scope-down, with a <code>NotStatement</code> around an IP set reference, combined with your path match in an <code>AndStatement</code>. The partner is exempt from counting and nothing else.</p>



<h2 class="wp-block-heading">Failure family 3: the response breaks the client instead of slowing it</h2>



<p class="wp-block-paragraph">The default Block action returns a 403. To a well-behaved API client, 403 means &#8220;you&#8217;re not allowed&#8221;, and many clients treat that as permanent: they stop, raise an alert, or worse, retry immediately in a loop that keeps them over the limit.</p>



<p class="wp-block-paragraph">A rate limit should say what it is. AWS WAF supports a custom response on Block, so you can return 429 with a <code>Retry-After</code> header, which HTTP clients and SDKs generally understand as &#8220;back off and try later&#8221;:</p>



<pre class="wp-block-code"><code>{
  "Action": {
    "Block": {
      "CustomResponse": {
        "ResponseCode": 429,
        "ResponseHeaders": [
          {
            "Name": "Retry-After",
            "Value": "60"
          }
        ]
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">You can also reference a custom response body defined on the web ACL, such as a small JSON error, so API clients get something parseable.</p>



<p class="wp-block-paragraph">What about Challenge and CAPTCHA? They&#8217;re genuinely useful on human-facing pages, because a real browser passes a silent challenge and a basic script doesn&#8217;t. On API endpoints they&#8217;re a trap. A mobile app, a server-to-server integration or a webhook sender can&#8217;t complete a browser challenge, so for them Challenge behaves like a block with a confusing response. Keep those actions to routes where you know a browser is on the other end.</p>



<h2 class="wp-block-heading">Failure family 4: expecting precision the service doesn&#8217;t promise</h2>



<p class="wp-block-paragraph">AWS is explicit that WAF rate limiting is meant to control high request rates, not to enforce exact quotas. It estimates the rate with an algorithm that weights recent requests more heavily, so it acts <em>near</em> your limit rather than exactly on it. There&#8217;s also lag both ways: a burst can run over the limit for a while before it&#8217;s caught, and a client that has slowed down can stay limited for a short time after. AWS&#8217;s own caveats page gives the typical delay as under 30 seconds, with longer possible.</p>



<p class="wp-block-paragraph">Two consequences follow:</p>



<ul class="wp-block-list">

<li><strong>Don&#8217;t use WAF to enforce billing quotas or contractual API limits.</strong> If &#8220;1,000 calls per customer per hour&#8221; is in a contract, count it in your application, where you control the number exactly and can show it to the customer. API Gateway usage plans help, but AWS describes their throttling and quotas as best effort too.</li>


<li><strong>Editing a live rule resets its counters.</strong> Changing the limit, window, key, forwarded IP config or scope-down resets the rule&#8217;s counts and can pause its rate limiting for up to a minute. Tuning during an active attack gives the attacker a free minute each time.</li>

</ul>



<h2 class="wp-block-heading">Rolling out AWS WAF rate-based rules without collateral damage</h2>



<p class="wp-block-paragraph">The order below is the one I reach for first, because each step produces the evidence the next step needs.</p>



<ol class="wp-block-list">

<li><strong>Turn on WAF logging before anything else.</strong> Send logs to CloudWatch Logs, S3 or Firehose. The destination name has to start with <code>aws-waf-logs-</code>. Without logs you&#8217;re tuning blind. If you want a longer retention and audit setup around them, that&#8217;s its own job: <a href="/aws/logging-audit/">AWS logging and audit trails</a>.</li>


<li><strong>Measure real traffic per key.</strong> Look at the busiest sources per five-minute bucket over a normal week, including your peak hour and any batch windows. The query further down does this in CloudWatch Logs Insights.</li>


<li><strong>Deploy every new rate rule in Count.</strong> Count mode evaluates the rule and records matches without affecting the request. Give it at least one full business cycle, ideally including a month-end or a marketing send.</li>


<li><strong>Review what would have been blocked.</strong> For each counted match, ask whether it was an attacker, a customer, a partner, a crawler you want, or your own monitoring. The last one catches people out: uptime checks and synthetic tests often run from a small pool of addresses.</li>


<li><strong>Set the limit above the busiest legitimate bucket, with headroom.</strong> The goal is to sit clearly between your heaviest real client and the flood, not to hug the average.</li>


<li><strong>Switch to Block with a 429 response,</strong> and add an alarm on the rule&#8217;s blocked request metric so a sudden jump reaches a human.</li>

</ol>



<p class="wp-block-paragraph">Here&#8217;s the Logs Insights query for step two, run against the WAF log group. It shows the heaviest client IPs per five-minute bin:</p>



<pre class="wp-block-code"><code>fields httpRequest.clientIp
| stats count(*) as requests by httpRequest.clientIp, bin(5m)
| sort requests desc
| limit 25</code></pre>



<p class="wp-block-paragraph">Swap <code>httpRequest.clientIp</code> for <code>httpRequest.uri</code> to find the hottest paths, or filter on a URI to size a scoped rule. At high volume, logs in S3 queried with Athena are usually cheaper to explore than repeated Logs Insights scans.</p>



<h2 class="wp-block-heading">Troubleshooting a customer who got rate limited</h2>



<p class="wp-block-paragraph">When the ticket arrives, work through it in this order:</p>



<ol class="wp-block-list">

<li><strong>Confirm it was WAF.</strong> A WAF block never reaches your application logs. Search WAF logs for the customer&#8217;s IP or API key and check <code>terminatingRuleId</code> and <code>action</code>.</li>


<li><strong>List what the rule is currently limiting.</strong> For rules aggregated only on IP or forwarded IP, AWS WAF can return the addresses currently being limited. The command is shown below. It won&#8217;t work for custom-key rules; for those, the logs are your only view.</li>


<li><strong>Check which address WAF saw.</strong> If it&#8217;s a CloudFront or CDN range, your rule is counting the proxy. If it&#8217;s the customer&#8217;s office NAT, the limit is fine for a person and wrong for an organization.</li>


<li><strong>Check for traffic you didn&#8217;t expect.</strong> A retry loop, a misconfigured cron job or a client polling far more often than it needs to will look exactly like abuse. Sometimes the fix is on their side, and the logs give you the evidence to say so politely.</li>


<li><strong>Fix the scope or key, not just the number.</strong> Raising the limit for everyone to fit one heavy customer weakens the rule for every attacker too.</li>

</ol>



<pre class="wp-block-code"><code>aws wafv2 get-rate-based-statement-managed-keys 
  --scope REGIONAL 
  --region eu-west-1 
  --web-acl-name my-web-acl 
  --web-acl-id 11111111-2222-3333-4444-555555555555 
  --rule-name rate-login</code></pre>



<p class="wp-block-paragraph">For a CloudFront web ACL, use <code>--scope CLOUDFRONT</code> with <code>--region us-east-1</code>. If the rule sits inside a rule group, add <code>--rule-group-rule-name</code> with the name of the rule group reference in your web ACL. A single rate-based rule instance limits at most 10,000 addresses at once; beyond that, the highest-rate addresses are the ones limited.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Putting an IP-keyed rate rule on an ALB behind a CDN and never checking which IP WAF sees.</li>


<li>Trusting the first address in <code>X-Forwarded-For</code> when nothing you control overwrites it.</li>


<li>Keying on a header or cookie with no companion rule for requests that don&#8217;t carry it.</li>


<li>One site-wide limit instead of scoped rules for login, expensive endpoints and the API.</li>


<li>Going straight to Block without a Count period on real traffic.</li>


<li>Returning 403 for a rate limit, or Challenge on endpoints non-browser clients call.</li>


<li>Exempting a partner with Allow and silently removing every protection below it.</li>


<li>Tweaking a live rule repeatedly during an incident and resetting its counters each time.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>Put IP-based rate rules on the web ACL closest to the real client, usually the CloudFront distribution.</li>


<li>Size limits from logged peak traffic per key, not from guesses about &#8220;normal&#8221; users.</li>


<li>Alarm on sudden changes in blocked requests per rule, in CloudWatch or whatever monitoring stack you already watch, such as Grafana or Datadog. <a href="/aws/monitoring-alerting/">Alerts that fire only when customers are affected</a> is the goal.</li>


<li>Keep contractual quotas in the application; use WAF for floods and abuse.</li>


<li>Consider AWS Shield Advanced only if you actually need its DDoS response support and cost protection; rate-based rules alone handle a lot of ordinary abuse.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is a rate-based rule in AWS WAF?</h3>



<p class="wp-block-paragraph">It&#8217;s a rule that counts requests per aggregation key (by default, per source IP) over a look-back window and applies an action, such as Block or Count, to keys that go over a limit you set. It&#8217;s aimed at floods, brute force and scraping rather than exact quota enforcement.</p>



<h3 class="wp-block-heading">Why is AWS WAF blocking legitimate users?</h3>



<p class="wp-block-paragraph">Usually because many real users share one key: an office NAT, carrier-grade NAT, a VPN exit node, or a CDN or load balancer whose address WAF sees instead of the client&#8217;s. Check which IP appears in the WAF log for the blocked request.</p>



<h3 class="wp-block-heading">Should I use IP or forwarded IP aggregation behind CloudFront?</h3>



<p class="wp-block-paragraph">Where you can, put the rule on the CloudFront web ACL and use plain IP aggregation, since WAF sees the viewer&#8217;s address there. Forwarded IP on an ALB uses the first address in the header, which the client can forge unless something you control overwrites it.</p>



<h3 class="wp-block-heading">How do I return 429 instead of 403 from AWS WAF?</h3>



<p class="wp-block-paragraph">Use a Block action with a custom response: set the response code to 429 and add a <code>Retry-After</code> response header. You can also attach a custom response body defined on the web ACL.</p>



<h3 class="wp-block-heading">Why didn&#8217;t my rate-based rule block at exactly the limit?</h3>



<p class="wp-block-paragraph">AWS WAF estimates request rates and doesn&#8217;t guarantee an exact limit match. There&#8217;s also a detection delay, and editing the rule resets its counts. For exact per-client quotas, count in the application.</p>



<h3 class="wp-block-heading">Can I see which clients a rate-based rule is currently blocking?</h3>



<p class="wp-block-paragraph">Yes, for rules that aggregate only on IP or forwarded IP, via <code>GetRateBasedStatementManagedKeys</code>. Rules with custom keys don&#8217;t support that call, so use WAF logs instead.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The one thing worth remembering about AWS WAF rate-based rules: the limit is the easy part. What decides if you block the attacker or your best customer is <em>what counts as one client</em>, and that choice is made by the aggregation key, the scope-down and where the web ACL sits. Log first, count before you block, give each kind of traffic its own rule, and answer with a 429 that tells well-behaved clients how to back off.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help tuning AWS WAF rate limits?</h2>



<p class="wp-block-paragraph">I help teams put rate limiting in front of AWS workloads that stops abuse and leaves real traffic alone. That usually covers:</p>



<ul class="wp-block-list">

<li>Reviewing an existing web ACL to find rules that count proxies, shared NATs or spoofable headers as single clients.</li>


<li>Moving IP-based rate rules to the right layer when CloudFront or another CDN sits in front of an ALB or API Gateway.</li>


<li>Designing scoped rules for login, signup, OTP and expensive endpoints, with composite keys for authenticated APIs.</li>


<li>Setting up WAF logging and the queries that size limits from your real peak traffic.</li>


<li>Running new rules in Count mode, reviewing what they&#8217;d catch, and switching to Block with 429 responses and alarms.</li>


<li>Untangling a live incident where customers or partners are being rate limited and you need to know why.</li>

</ul>



<p class="wp-block-paragraph">If a customer is getting blocked right now, send me the rule JSON and one redacted WAF log entry for the blocked request. That&#8217;s usually enough to see which of the four failure families you&#8217;re in.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/aws/waf-protection/">See the AWS WAF protection service</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/">AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Plenty of CPU, Zero Connections: Debugging Connection Pool Exhaustion on AWS</title>
		<link>https://john-nessime.com/blog/cloud-computing/connection-pool-exhaustion-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 08 Oct 2026 06:17:49 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Site Reliability Engineering]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Connection Leaks]]></category>
		<category><![CDATA[Connection Pinning]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[HikariCP]]></category>
		<category><![CDATA[Idle in Transaction]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[max_connections]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Performance Tuning]]></category>
		<category><![CDATA[PgBouncer]]></category>
		<category><![CDATA[Pool Exhaustion]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[RDS Proxy]]></category>
		<category><![CDATA[SRE]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1146</guid>

					<description><![CDATA[<p>Pool timeouts while RDS sits at low CPU? The database isn't the bottleneck. Here's how to tell whether your app pool, max_connections or RDS Proxy ran out, the SQL that proves it, and the fixes that survive the next deploy.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/connection-pool-exhaustion-aws/">Plenty of CPU, Zero Connections: Debugging Connection Pool Exhaustion on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The alert says p99 latency is climbing. The application logs are full of <code>Connection is not available, request timed out after 30000ms</code>. Someone opens the RDS console, looks at CPU sitting low, freeable memory looking healthy and <code>DatabaseConnections</code> nowhere near the limit, and says the obvious thing: &#8220;The database is fine.&#8221;</p>



<p class="wp-block-paragraph">They&#8217;re right. The database is fine. That&#8217;s exactly why this incident drags on for an hour instead of ten minutes.</p>



<p class="wp-block-paragraph">Connection pool exhaustion on AWS is rarely a capacity problem. It&#8217;s a queueing problem, and the queue forms in one of three places: inside your application&#8217;s pool, at the database&#8217;s connection limit, or inside a proxy sitting between the two. Each layer has its own limit, its own error message and its own fix. Resizing the instance fixes none of them reliably, and sometimes makes one worse.</p>



<p class="wp-block-paragraph">Below: how to tell which layer ran out, the SQL and metrics that prove it, and fixes that survive the next deploy. Examples use RDS and Aurora for PostgreSQL and MySQL, plus HikariCP, SQLAlchemy and node-postgres where their defaults matter.</p>



<h2 class="wp-block-heading">First question: which layer ran out?</h2>



<p class="wp-block-paragraph">Before touching anything, figure out where requests are waiting. The error text usually tells you, if you read it literally instead of pattern-matching on &#8220;connection&#8221;.</p>



<ul class="wp-block-list">
<li><strong>The application pool is empty.</strong> The error comes from your pool library, not the database. HikariCP says the connection is not available and the request timed out. SQLAlchemy raises a <code>QueuePool limit ... reached</code> timeout. The database never saw these requests at all.</li>



<li><strong>The database refused new connections.</strong> The error comes from the server. PostgreSQL returns a <code>FATAL</code> about too many clients or about the remaining connection slots being reserved (the exact wording varies by major version). MySQL returns error 1040, <code>Too many connections</code>.</li>



<li><strong>The proxy is holding the queue.</strong> With RDS Proxy in the path, clients connect fine but wait for a database connection to be borrowed, then time out after the proxy&#8217;s borrow timeout.</li>
</ul>



<p class="wp-block-paragraph">Then compare what the database is actually holding against its limit. On PostgreSQL:</p>



<pre class="wp-block-code"><code>SHOW max_connections;

SELECT count(*) AS client_connections
FROM pg_stat_activity
WHERE backend_type = 'client backend';</code></pre>



<p class="wp-block-paragraph">On MySQL, <code>Max_used_connections</code> is the high-water mark since the last restart, which is useful when the spike has already passed:</p>



<pre class="wp-block-code"><code>SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE 'Threads_connected';
SHOW GLOBAL STATUS LIKE 'Max_used_connections';</code></pre>



<p class="wp-block-paragraph">If the app is throwing pool timeouts while the server is well under its limit, you&#8217;re in the first failure family. If the server is at the limit, it&#8217;s the second. That single comparison saves you from the most common wrong move, which is raising <code>max_connections</code> when the database was never the bottleneck.</p>



<h2 class="wp-block-heading">Pool exhaustion with an idle database: the one that hides</h2>



<p class="wp-block-paragraph">This is the failure that fools people, because every database graph looks healthy while the application is effectively down.</p>



<p class="wp-block-paragraph">A pool hands out a fixed number of connections per process. A request borrows one, and nobody else can use it until it&#8217;s returned. The pool doesn&#8217;t care whether the request is running SQL during that time. If your code opens a transaction, runs a query, then calls a payment API, uploads to S3 or sends an email before committing, the connection sits borrowed and idle for the whole round trip.</p>



<p class="wp-block-paragraph">Now let that third-party API slow down. Each request holds its connection longer, the pool drains, new requests queue for a connection, and they time out. The database sees a handful of sessions doing nothing. CPU drops, because less SQL is running than usual. The dashboard gets greener as the outage gets worse.</p>



<p class="wp-block-paragraph">So <strong>a slow dependency that has nothing to do with the database can exhaust your database pool</strong>, and database metrics will never show it.</p>



<h3 class="wp-block-heading">Prove it with pg_stat_activity</h3>



<p class="wp-block-paragraph">Group sessions by state and by who owns them. The <code>state</code> column is the evidence; <code>idle in transaction</code> means a client opened a transaction and then went quiet without committing.</p>



<pre class="wp-block-code"><code>SELECT state,
       application_name,
       count(*) AS sessions,
       max(now() - xact_start) AS oldest_transaction
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY state, application_name
ORDER BY sessions DESC;</code></pre>



<p class="wp-block-paragraph">Then list the worst offenders. The <code>query</code> column shows the last statement the session ran, not what it&#8217;s doing now, which is exactly what you want here: it tells you where in the code the transaction was left open.</p>



<pre class="wp-block-code"><code>SELECT pid,
       application_name,
       client_addr,
       now() - xact_start AS transaction_age,
       left(query, 80) AS last_statement
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY xact_start;</code></pre>



<p class="wp-block-paragraph">On MySQL, open transactions live in <code>information_schema.INNODB_TRX</code>. The thread ID joins back to the process list so you can see the host and user:</p>



<pre class="wp-block-code"><code>SELECT trx_mysql_thread_id, trx_started, trx_state
FROM information_schema.INNODB_TRX
ORDER BY trx_started;</code></pre>



<p class="wp-block-paragraph">One thing makes all of this much faster: set <code>application_name</code> per service in the connection string. Without it, every session looks the same and you&#8217;re guessing from IP addresses that change with every task restart.</p>



<h3 class="wp-block-heading">Leak or slow holder?</h3>



<p class="wp-block-paragraph">Two different bugs produce the same symptom. A <strong>leak</strong> is a code path that borrows a connection and never returns it, usually an exception path that skips the close. A <strong>slow holder</strong> returns the connection eventually but keeps it through non-database work.</p>



<p class="wp-block-paragraph">The tell is recovery. Leaks never recover without a restart; the pool ratchets down one connection at a time until it&#8217;s empty. Slow holders recover when the slow dependency recovers. If restarting the service &#8220;fixes&#8221; the incident every few days, suspect a leak.</p>



<p class="wp-block-paragraph">HikariCP has <code>leakDetectionThreshold</code>, which is off by default. Set it to a value longer than your slowest legitimate request, and it logs the stack trace of wherever a connection was borrowed and held past that threshold. That stack trace is usually the whole diagnosis. Other pools have their own hooks; if yours doesn&#8217;t, log borrow duration yourself around the checkout.</p>



<h3 class="wp-block-heading">Fixes that actually hold</h3>



<ol class="wp-block-list">
<li><strong>Stop holding connections across network calls.</strong> Do the external call first, then open the transaction, write, and commit. If you need the call&#8217;s result inside the transaction, restructure so the transaction is short. This is the real fix; everything else is a guardrail.</li>



<li><strong>Put a server-side ceiling on idle transactions.</strong> PostgreSQL&#8217;s <code>idle_in_transaction_session_timeout</code> terminates a session that sits idle inside a transaction past the limit. The app will see an error, and that&#8217;s the point: a loud failure beats a silent pool drain. You can set it in the parameter group or scope it to the application role.</li>



<li><strong>Fail fast on checkout.</strong> A web request that waits 30 seconds for a connection is worse than one that fails in two. Match the pool&#8217;s acquisition timeout to what the caller will tolerate.</li>
</ol>



<pre class="wp-block-code"><code>ALTER ROLE app_user SET idle_in_transaction_session_timeout = '60s';</code></pre>



<p class="wp-block-paragraph">The role-level setting applies to new sessions for that role, so existing pooled connections keep the old value until they&#8217;re recycled. Pick a limit comfortably above your longest legitimate pause, or you&#8217;ll kill batch jobs that were behaving.</p>



<p class="wp-block-paragraph">MySQL has no direct equivalent. <code>wait_timeout</code> closes idle sessions, but it&#8217;s a blunt setting that also affects pooled connections sitting legitimately idle, so tune it alongside your pool&#8217;s own idle and max-lifetime settings rather than instead of fixing the code.</p>



<p class="wp-block-paragraph">Watch the node-postgres default specifically. Its pool <code>connectionTimeoutMillis</code> defaults to 0, which means no timeout: when the pool is empty, requests wait forever. You don&#8217;t get an error to alert on; you get hung requests and upstream timeouts that point everywhere except the pool.</p>



<h2 class="wp-block-heading">Server slots run out while CPU idles</h2>



<p class="wp-block-paragraph">The second family is the one people expect: the database refuses connections. The surprise is that it happens at low CPU, and that the cause is arithmetic, not load.</p>



<h3 class="wp-block-heading">Pool multiplication</h3>



<p class="wp-block-paragraph">Every pool size is per process. The number the database actually faces is the product of everything that opens a pool:</p>



<pre class="wp-block-code"><code>worst case = pool size
           x worker processes per container
           x containers at the autoscaler's maximum
           x 2 during a rolling deploy (old and new running together)
           + cron jobs, migrations, BI tools, admin sessions</code></pre>



<p class="wp-block-paragraph">Take an illustrative case: a pool of 10, four worker processes per ECS task, 30 tasks at the scaling maximum. That&#8217;s 1,200 potential connections before a deploy briefly doubles it. Nobody chose that number. It emerged from three settings owned by three different people, and it only shows up on the day traffic and a deploy coincide.</p>



<p class="wp-block-paragraph">Use the autoscaler&#8217;s maximum, not today&#8217;s task count. Today&#8217;s count is what makes the math look safe.</p>



<h3 class="wp-block-heading">The limit moves when you resize</h3>



<p class="wp-block-paragraph">On RDS and Aurora, the default <code>max_connections</code> in the parameter group is a formula based on the instance class&#8217;s memory, not a fixed number. So a rightsizing exercise that drops the instance a size to save money also quietly lowers the connection ceiling. The CPU graph said it was safe. The connection budget didn&#8217;t get a vote.</p>



<p class="wp-block-paragraph">Don&#8217;t calculate the value by hand; the memory figure RDS uses excludes what its own processes consume. Run <code>SHOW max_connections</code> on the actual instance, and re-run it after every class change.</p>



<h3 class="wp-block-heading">Why raising max_connections is the wrong reflex</h3>



<p class="wp-block-paragraph">You can override <code>max_connections</code> in a custom parameter group, and sometimes a modest bump is fine. But it treats the symptom. Each PostgreSQL connection is a separate backend process with its own memory, and sorts or hashes inside it can each use up to <code>work_mem</code>. Thousands of connections all trying to work at once contend for the same CPU cores, and throughput often drops as concurrency rises past what the hardware can run in parallel.</p>



<p class="wp-block-paragraph">Also keep in mind that the last few slots are reserved for administrative roles. The application starts failing before the counter reaches the limit, and when it does, you may struggle to get your own session in to diagnose it. Keep headroom for humans.</p>



<h3 class="wp-block-heading">Budget connections on purpose</h3>



<ol class="wp-block-list">
<li>Read the real <code>max_connections</code> from the instance and subtract a margin for admin sessions, monitoring agents and migrations.</li>



<li>List every consumer: each service, its worker count, its scaling maximum, and its batch jobs.</li>



<li>Divide the budget and set each pool&#8217;s maximum from that, including the deploy overlap.</li>



<li>Cap the autoscaler maximum at something the database can serve. An autoscaler that can add tasks faster than the database can accept connections just converts a traffic spike into a connection storm.</li>



<li>If the budget can&#8217;t cover the fleet, add a pooler in front instead of more slots.</li>
</ol>



<p class="wp-block-paragraph">Smaller pools feel scary, but a connection is only useful while SQL is running. A small pool with short transactions usually beats a large pool with long ones.</p>



<h3 class="wp-block-heading">Zombie connections</h3>



<p class="wp-block-paragraph">Some slots belong to clients that are gone. A container killed without a clean shutdown, or a Lambda execution environment frozen between invocations, can leave server-side sessions open until something times them out. With Lambda, each concurrent environment holds its own connection, so cap concurrency and put a pooler in front; the <a href="https://john-nessime.com/aws/serverless-api/">serverless API</a> work I do starts exactly there.</p>



<p class="wp-block-paragraph">For long-lived services, set the pool&#8217;s idle timeout and max lifetime below any idle timeout on the network path, so connections are retired by the pool rather than cut silently in the middle.</p>



<h2 class="wp-block-heading">When the proxy becomes the queue</h2>



<p class="wp-block-paragraph">RDS Proxy is the managed answer to pool multiplication. Many client connections share a smaller set of database connections, and the proxy hands a database connection to a client per transaction. When that works, the server-side limit stops being your problem.</p>



<p class="wp-block-paragraph">It stops working in two ways. First, <strong>pinning</strong>: when a session changes state the proxy can&#8217;t safely share, such as setting a variable on PostgreSQL, the proxy ties that client to one database connection until the session ends. Enough pinned sessions and the proxy behaves like no proxy at all. Second, the <strong>borrow queue</strong>: when every database connection the proxy is allowed to open is busy, clients wait up to <code>ConnectionBorrowTimeout</code> (120 seconds by default) and then fail. Your application sees slow requests, not refused connections, and the database graphs look calm again.</p>



<p class="wp-block-paragraph">The proxy&#8217;s own CloudWatch metrics tell you which it is. <code>DatabaseConnectionsCurrentlySessionPinned</code> shows pinning. <code>DatabaseConnectionsCurrentlyBorrowed</code> against <code>DatabaseConnections</code> shows how close the proxy is to its share of the database. And <code>MaxConnectionsPercent</code> on the target group caps that share as a percentage of <code>max_connections</code>, so a conservative setting there creates the borrow queue by itself.</p>



<p class="wp-block-paragraph">The proxy is billed on the capacity of the database behind it (per vCPU-hour for provisioned instances, per ACU-hour for Aurora Serverless), so run your current rates through something like the <a href="https://john-nessime.com/tools/aws-cost-estimator/">AWS Cost Estimator</a> before assuming it&#8217;s free relief. PgBouncer in transaction mode on a small EC2 instance is the self-managed alternative: cheaper to run, but it&#8217;s one more thing to patch, monitor and make highly available.</p>



<h2 class="wp-block-heading">Metrics that warn you before connection pool exhaustion</h2>



<p class="wp-block-paragraph">Database CPU is the wrong alarm for this problem. By the time connection pressure shows up there, if it ever does, users already noticed. The signals that lead are on the client side:</p>



<ul class="wp-block-list">
<li><strong>Threads waiting for a connection.</strong> HikariCP exposes pending threads, active and idle counts, and acquisition time through its Micrometer integration. Sustained pending threads above zero is the earliest reliable warning you get.</li>



<li><strong>Connection acquisition time.</strong> Healthy pools hand out connections almost instantly. A rising acquisition time is the queue forming, long before timeouts start.</li>



<li><strong>Server connections as a ratio of the limit.</strong> Alarm on CloudWatch <code>DatabaseConnections</code> relative to the instance&#8217;s real <code>max_connections</code>, not an absolute number that goes stale after a resize.</li>



<li><strong>Oldest open transaction.</strong> A scheduled query against <code>pg_stat_activity</code> exporting the age of the oldest <code>xact_start</code> catches slow holders and leaks early.</li>
</ul>



<p class="wp-block-paragraph">Where these land matters less than having them next to each other. Grafana with a Prometheus scrape of the application&#8217;s pool metrics works well, as do Datadog or New Relic APM, which can show pool wait time inside the request trace. Sentry is useful for grouping the pool-timeout exceptions so you see the first one, not the thousandth. The goal is one screen where app pool state and database connection counts share a time axis. If you&#8217;d rather not build that, it&#8217;s the kind of thing covered under <a href="https://john-nessime.com/aws/monitoring-alerting/">AWS monitoring and alerting</a>.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<ul class="wp-block-list">
<li><strong>Pool timeouts, database far below its limit, many idle-in-transaction sessions:</strong> slow holder or leak. Find the last statement in <code>pg_stat_activity</code> and trace the code path that follows it.</li>



<li><strong>Pool timeouts, few database sessions at all:</strong> the pool is too small for the worker count, or connections are borrowed and held without a transaction open. Check borrow duration, not the database.</li>



<li><strong>Refused connections only during deploys:</strong> the old and new versions overlap. Budget for the overlap or slow the rollout.</li>



<li><strong>Refused connections right after an instance class change:</strong> the formula-derived limit dropped. Check <code>SHOW max_connections</code>.</li>



<li><strong>Requests hang with no error at all:</strong> a pool with no acquisition timeout, such as node-postgres at its default. Set one.</li>



<li><strong>Everything was fixed by a restart, then came back days later:</strong> a leak. Turn on leak detection and wait for the stack trace.</li>



<li><strong>RDS Proxy in the path, slow requests, calm database:</strong> check pinned and borrowed connection metrics before anything else.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Scaling the instance up because &#8220;it&#8217;s the database&#8221;, when the database was idle the whole time.</li>



<li>Raising <code>max_connections</code> to absorb pool multiplication, then losing throughput to contention.</li>



<li>Making the pool bigger to fix pool timeouts caused by long transactions. It buys minutes and moves the failure to the server limit.</li>



<li>Sizing pools from today&#8217;s task count instead of the autoscaler&#8217;s maximum.</li>



<li>Adding RDS Proxy without checking pinning, and ending up with a paid queue instead of a pool.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep transactions short and never hold a connection across an external call.</li>



<li>Write the connection budget down, with every consumer, and revisit it when scaling limits or instance classes change.</li>



<li>Set an acquisition timeout that matches what callers will tolerate, and alert on pending threads, not on timeouts.</li>



<li>Use <code>idle_in_transaction_session_timeout</code> on PostgreSQL as a safety net, scoped to application roles.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What causes connection pool exhaustion when database CPU is low?</h3>



<p class="wp-block-paragraph">Usually connections held while no SQL runs: long transactions, external API calls inside a transaction, or leaked connections that were never returned. The pool is empty, but the database is doing very little.</p>



<h3 class="wp-block-heading">Should I increase max_connections on RDS?</h3>



<p class="wp-block-paragraph">Only after confirming the server limit is actually the one being hit, and only modestly. If the cause is pool multiplication, budget the pools or add a pooler. More connections means more memory per backend and more contention, not more throughput.</p>



<h3 class="wp-block-heading">How big should my connection pool be?</h3>



<p class="wp-block-paragraph">Smaller than most people set it. Start from the connection budget divided across every process at maximum scale, then load test. A pool large enough to cover every thread usually just moves the queue from the application to the database.</p>



<h3 class="wp-block-heading">Does RDS Proxy fix pool exhaustion?</h3>



<p class="wp-block-paragraph">It fixes server-side slot exhaustion from many clients, as long as sessions aren&#8217;t pinned. It does nothing for an application pool drained by long transactions, because those transactions hold the proxy&#8217;s database connections just as firmly.</p>



<h3 class="wp-block-heading">How do I find leaked connections in PostgreSQL?</h3>



<p class="wp-block-paragraph">Look for sessions in <code>pg_stat_activity</code> that stay idle or idle in transaction for far longer than any request should, grouped by <code>application_name</code>. Then turn on your pool&#8217;s leak detection to get the stack trace of where the connection was borrowed.</p>



<h3 class="wp-block-heading">Does this happen on self-hosted databases too?</h3>



<p class="wp-block-paragraph">Exactly the same way. On PostgreSQL running on a VPS from providers like Hetzner, Contabo or InterServer, the pool math and idle transactions behave identically; the difference is that you run PgBouncer yourself instead of paying for RDS Proxy.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Connection pool exhaustion on AWS almost never means the database is too small. It means a queue formed somewhere, and the fix depends entirely on where. Read the error literally, compare the database&#8217;s connection count against its real limit, and within minutes you&#8217;ll know if you&#8217;re dealing with a drained application pool, a server limit reached by arithmetic, or a proxy holding the line.</p>



<p class="wp-block-paragraph">If you remember one thing: a connection costs you for as long as it&#8217;s borrowed, not for as long as SQL runs. Shorten the borrow and most of these incidents stop happening.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help tracking down a connection problem on AWS?</h2>



<p class="wp-block-paragraph">This is work I do for teams running RDS, Aurora or self-managed PostgreSQL and MySQL. Typical engagements look like this:</p>



<ul class="wp-block-list">
<li>Working out which layer is exhausting during an incident, from pool logs, <code>pg_stat_activity</code> and CloudWatch.</li>



<li>Building a written connection budget across services, workers, autoscaling limits and deploy overlap.</li>



<li>Finding leaks and long-held connections in application code, with leak detection and transaction timeouts set sensibly.</li>



<li>Evaluating RDS Proxy or PgBouncer for your workload, including pinning and cost.</li>



<li>Dashboards and alerts that put application pool state and database connections on the same screen.</li>



<li>Checking what an instance resize will do to your connection limit before you make it.</li>
</ul>



<p class="wp-block-paragraph">Send me the pool timeout from your logs and the output of the <code>pg_stat_activity</code> query above, and I&#8217;ll tell you which layer ran out.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/hosted-monitoring/">See hosted monitoring</a></div>



<div class="wp-block-button is-style-outline is-style-outline--2"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/connection-pool-exhaustion-aws/">Plenty of CPU, Zero Connections: Debugging Connection Pool Exhaustion on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</title>
		<link>https://john-nessime.com/blog/devops/monitoring-zoho-integrations/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 06 Oct 2026 06:08:51 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<category><![CDATA[Zoho Flow]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=395</guid>

					<description><![CDATA[<p>Zoho integrations rarely fail loudly. Channels expire after an hour, credits drain against a rolling window, and bulk writes return HTTP 200 while rejecting records. Here is how to instrument for absence using CloudWatch Embedded Metric Format, Loki's absent_over_time, and Grafana across both.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message came in on a Tuesday: &#8220;Have we stopped getting leads from the website form?&#8221; Not an alert. Not a page. A salesperson noticing the pipeline looked thin.</p>



<p class="wp-block-paragraph">The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.</p>



<p class="wp-block-paragraph">That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.</p>



<h2 class="wp-block-heading">The three ways Zoho integrations actually break</h2>



<p class="wp-block-paragraph">Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:</p>



<ol class="wp-block-list">
<li><strong>Silent stop.</strong> Zoho stops sending you events. Nothing fails, because nothing happens.</li>

<li><strong>Silent throttle.</strong> You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.</li>

<li><strong>Silent drift.</strong> Calls succeed, records do not land, and the two systems diverge without a single error log.</li>
</ol>



<p class="wp-block-paragraph">An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.</p>



<h2 class="wp-block-heading">Failure family one: the integration that just stops</h2>



<p class="wp-block-paragraph">If you push data out of Zoho CRM using the Notifications API, you subscribe a <code>notify_url</code> to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.</p>



<p class="wp-block-paragraph">Notification channels expire. Zoho&#8217;s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is <strong>one hour</strong>. So the failure mode is not &#8220;someone forgot to renew after a year.&#8221; It is &#8220;someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.&#8221;</p>



<p class="wp-block-paragraph">What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for <em>absence</em>.</p>



<h3 class="wp-block-heading">Emit a heartbeat, then alarm on its absence</h3>



<p class="wp-block-paragraph">The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No <code>PutMetricData</code> call, no synchronous latency in your request path.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}</code></pre>



<p class="wp-block-paragraph">Note what is a dimension and what is not. <code>Module</code> and <code>Channel</code> are bounded sets, so they are safe dimensions. <code>record_id</code> sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.</p>



<p class="wp-block-paragraph">Now the alarm. A normal CloudWatch alarm goes to <code>INSUFFICIENT_DATA</code> when the metric stops reporting, and <code>INSUFFICIENT_DATA</code> does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts</code></pre>



<p class="wp-block-paragraph">The <code>--treat-missing-data breaching</code> flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.</p>



<p class="wp-block-paragraph">On the Loki side the equivalent is <code>absent_over_time</code>, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:</p>



<pre class="wp-block-code"><code>groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>absent_over_time</code> only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.</p>



<p class="wp-block-paragraph">Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit <code>ChannelSecondsRemaining</code> at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.</p>



<h2 class="wp-block-heading">Failure family two: throttling that is not rate limiting</h2>



<p class="wp-block-paragraph">Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.</p>



<p class="wp-block-paragraph">Zoho CRM meters API usage in <strong>credits</strong>, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.</p>



<h3 class="wp-block-heading">The gauge that appears halfway through the tank</h3>



<p class="wp-block-paragraph">Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an <code>X-API-CREDITS-REMAINING</code> response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.</p>



<p class="wp-block-paragraph">Read that again in monitoring terms. An absent header means either &#8220;you are comfortably under half your budget&#8221; or &#8220;your instrumentation is broken and you have no idea.&#8221; Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:</p>



<ul class="wp-block-list">
<li>A counter for responses where the header was <em>present</em>, which effectively tells you when you crossed the halfway mark.</li>

<li>A gauge of the value itself, which only has meaning while the header exists.</li>

<li>Your own client-side count of calls made, which works regardless and is the only thing you control.</li>
</ul>



<p class="wp-block-paragraph">The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.</p>



<h3 class="wp-block-heading">Concurrency is a separate ceiling</h3>



<p class="wp-block-paragraph">Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns <code>TOO_MANY_REQUESTS</code>.</p>



<p class="wp-block-paragraph">Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a <code>finally</code> block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.</p>



<p class="wp-block-paragraph">One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone&#8217;s workflow rule is spending your budget. Zoho&#8217;s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.</p>



<h2 class="wp-block-heading">Failure family three: successful calls that did nothing</h2>



<p class="wp-block-paragraph">Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.</p>



<p class="wp-block-paragraph">If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.</p>



<p class="wp-block-paragraph">The fix is to stop counting requests and start counting records, on both sides:</p>



<ul class="wp-block-list">
<li><strong>Records submitted</strong> versus <strong>records confirmed</strong> in the per-record response status.</li>

<li>A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.</li>

<li>A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.</li>
</ul>



<p class="wp-block-paragraph">In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:</p>



<pre class="wp-block-code"><code>sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))</code></pre>



<p class="wp-block-paragraph">A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.</p>



<h2 class="wp-block-heading">Deciding which signal goes where</h2>



<p class="wp-block-paragraph">Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:</p>



<ul class="wp-block-list">
<li><strong>CloudWatch</strong> holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.</li>

<li><strong>Loki</strong> holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.</li>

<li><strong>Grafana</strong> is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.</li>
</ul>



<p class="wp-block-paragraph">The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.</p>



<p class="wp-block-paragraph">Loki&#8217;s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:</p>



<pre class="wp-block-code"><code>limits_config:
  allow_structured_metadata: true
  volume_enabled: true</code></pre>



<p class="wp-block-paragraph">Put <code>module</code>, <code>environment</code> and <code>job</code> in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.</p>



<p class="wp-block-paragraph">On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.</p>



<p class="wp-block-paragraph">Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration&#8217;s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.</p>



<h2 class="wp-block-heading">Troubleshooting the monitoring itself</h2>



<ul class="wp-block-list">
<li><strong>EMF metrics never appear.</strong> The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.</li>

<li><strong>The alarm never fires despite obvious silence.</strong> Look at the alarm state. Sitting in <code>INSUFFICIENT_DATA</code> means the missing-data treatment was left at its default.</li>

<li><strong><code>absent_over_time</code> returns nothing.</strong> That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.</li>

<li><strong>Loki queries slow to a crawl.</strong> Check stream count before blaming resources. A label added &#8220;temporarily&#8221; during debugging is the usual cause.</li>

<li><strong>Webhook deliveries stop after a deployment.</strong> Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.</li>

<li><strong>Credit consumption does not match your call count.</strong> Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on error rate alone, which is blind to every failure described above.</li>

<li>Creating notification channels without an explicit <code>channel_expiry</code> and inheriting the one-hour default.</li>

<li>Assuming an absent <code>X-API-CREDITS-REMAINING</code> header means healthy usage rather than sub-threshold usage.</li>

<li>Building a daily credit dashboard when the limit is a rolling window.</li>

<li>Putting record IDs into EMF dimensions or Loki labels.</li>

<li>Treating an HTTP 200 on a bulk write as confirmation that every record landed.</li>

<li>Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.</li>
</ul>



<h2 class="wp-block-heading">Best practices for monitoring Zoho integrations</h2>



<ol class="wp-block-list">
<li>Start with a deadman alert per channel. If you build one thing, build this.</li>

<li>Set <code>channel_expiry</code> explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.</li>

<li>Count records, not requests, on both sides of every write.</li>

<li>Keep a client-side call counter so you are never dependent on a header that may not be sent.</li>

<li>Gauge in-flight requests to see concurrency pressure before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.</li>

<li>Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.</li>

<li>Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why did my Zoho webhook stop working on its own?</h3>



<p class="wp-block-paragraph">Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.</p>



<h3 class="wp-block-heading">Can I list my active Zoho notification channels?</h3>



<p class="wp-block-paragraph">There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho&#8217;s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.</p>



<h3 class="wp-block-heading">Do I need both CloudWatch and Loki?</h3>



<p class="wp-block-paragraph">No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.</p>



<h3 class="wp-block-heading">How do I monitor Zoho API credits properly?</h3>



<p class="wp-block-paragraph">Record the <code>X-API-CREDITS-REMAINING</code> header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.</p>



<h3 class="wp-block-heading">What causes TOO_MANY_REQUESTS if I am under my credit limit?</h3>



<p class="wp-block-paragraph">Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.</p>



<h3 class="wp-block-heading">Should I still use Promtail for shipping integration logs?</h3>



<p class="wp-block-paragraph">No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.</p>



<h3 class="wp-block-heading">How long should the deadman alert window be?</h3>



<p class="wp-block-paragraph">Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.</p>



<p class="wp-block-paragraph">So invert the default. Alarm when the count is too low rather than when the errors are too high. Set <code>--treat-missing-data breaching</code>. Use <code>absent_over_time</code>. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting a Zoho integration?</h2>



<p class="wp-block-paragraph">Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:</p>



<ul class="wp-block-list">
<li>Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.</li>

<li>Instrumenting credit and concurrency headroom so throttling is visible before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.</li>

<li>Adding reconciliation jobs that compare record counts across systems and surface drift early.</li>

<li>Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.</li>

<li>Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</title>
		<link>https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 04 Oct 2026 06:06:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Amazon EKS]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Blue/Green Deployments]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cluster Autoscaler]]></category>
		<category><![CDATA[Cluster Insights]]></category>
		<category><![CDATA[CoreDNS]]></category>
		<category><![CDATA[Deprecated APIs]]></category>
		<category><![CDATA[Drift Detection]]></category>
		<category><![CDATA[EKS Auto Mode]]></category>
		<category><![CDATA[Graceful Shutdown]]></category>
		<category><![CDATA[Karpenter]]></category>
		<category><![CDATA[Node Autoscaling]]></category>
		<category><![CDATA[Node Lifecycle]]></category>
		<category><![CDATA[Pod Disruption Budgets]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Rolling Updates]]></category>
		<category><![CDATA[Runbooks]]></category>
		<category><![CDATA[VPC CNI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=839</guid>

					<description><![CDATA[<p>The control plane bump is the part AWS handles and almost never breaks. The damage happens in the data plane rollout afterwards. A practical walk through the three failure families that actually stall EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the node drain, plus what each escape hatch really costs.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The control plane upgrade finished in about twenty minutes and nobody noticed. Then the managed node group update went to <code>Updating</code>, sat there, and came back with <code>PodEvictionFailure</code>. Half the group was on the new AMI and half on the old, and the thing blocking the drain was a PodDisruptionBudget somebody wrote eighteen months earlier for a two-replica deployment.</p>



<p class="wp-block-paragraph">That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.</p>



<p class="wp-block-paragraph">This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.</p>



<h2 class="wp-block-heading">Why the control plane is the easy half</h2>



<p class="wp-block-paragraph">An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.</p>



<p class="wp-block-paragraph">Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the API calls you did not know you were making</h2>



<p class="wp-block-paragraph">Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.</p>



<p class="wp-block-paragraph">Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.</p>



<h3 class="wp-block-heading">Cluster insights, and what it is actually reading</h3>



<pre class="wp-block-code"><code>aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111</code></pre>



<p class="wp-block-paragraph">EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in <code>ERROR</code> until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:</p>



<pre class="wp-block-code"><code>aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster</code></pre>



<p class="wp-block-paragraph">The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.</p>



<h3 class="wp-block-heading">Ask the API server directly</h3>



<p class="wp-block-paragraph">The API server keeps its own counter, and it names the release that removes each API. Fastest check available:</p>



<pre class="wp-block-code"><code>kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis</code></pre>



<p class="wp-block-paragraph">Read the <code>removed_release</code> label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.</p>



<p class="wp-block-paragraph">For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:</p>



<pre class="wp-block-code"><code>fields @message
| filter `annotations.k8s.io/deprecated` = "true"</code></pre>



<p class="wp-block-paragraph">The <code>user.username</code> field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart&#8217;s owner. Static scanners like <code>kubent</code> and <code>pluto</code> cover the other half, the manifests that have not run recently. Use both.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: add-ons and the conflict resolution you picked once</h2>



<p class="wp-block-paragraph">Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.</p>



<p class="wp-block-paragraph">When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:</p>



<ul class="wp-block-list">
<li><code>NONE</code> leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.</li>



<li><code>OVERWRITE</code> reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.</li>



<li><code>PRESERVE</code> keeps your value through the update.</li>
</ul>



<p class="wp-block-paragraph"><code>OVERWRITE</code> is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with <code>OVERWRITE</code> will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.</p>



<pre class="wp-block-code"><code>aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version &lt;compatible-version&gt; 
    --resolve-conflicts PRESERVE</code></pre>



<p class="wp-block-paragraph">One more trap: the same value means different things on create and on update. On create, <code>PRESERVE</code> behaves close to <code>NONE</code>, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.</p>



<p class="wp-block-paragraph">The durable fix is to stop editing add-on resources with <code>kubectl</code> at all. Put customizations in the add-on&#8217;s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: PDB deadlocks and the fifteen-minute drain clock</h2>



<p class="wp-block-paragraph">This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:</p>



<ol class="wp-block-list">
<li>Pick a node to replace, up to the configured <code>maxUnavailable</code>.</li>



<li>Cordon it once a new node reaches <code>Ready</code>.</li>



<li>Drain the pods off it.</li>



<li>Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.</li>



<li>Repeat until nothing is left on the old launch template version.</li>
</ol>



<p class="wp-block-paragraph">Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with <code>PodEvictionFailure</code> and you are left with a half-migrated node group.</p>



<h3 class="wp-block-heading">The deadlock nobody sees coming</h3>



<p class="wp-block-paragraph">A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.</p>



<p class="wp-block-paragraph">One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:</p>



<pre class="wp-block-code"><code>kubectl get pdb --all-namespaces</code></pre>



<p class="wp-block-paragraph">Read the <code>ALLOWED DISRUPTIONS</code> column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.</p>



<p class="wp-block-paragraph">There is a second cause of <code>PodEvictionFailure</code> with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates <code>NoSchedule</code> unconditionally.</p>



<h3 class="wp-block-heading">What the force flag actually costs</h3>



<pre class="wp-block-code"><code>aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force</code></pre>



<p class="wp-block-paragraph">Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.</p>



<p class="wp-block-paragraph">My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.</p>



<h3 class="wp-block-heading">If you run Karpenter, the levers are different</h3>



<p class="wp-block-paragraph">Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node&#8217;s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:</p>



<pre class="wp-block-code"><code>spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h</code></pre>



<p class="wp-block-paragraph">Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not &#8220;no disruption&#8221;, they are &#8220;unlimited disruption&#8221;.</p>



<p class="wp-block-paragraph">The other lever, <code>terminationGracePeriod</code> on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the <code>karpenter.sh/do-not-disrupt</code> annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The rollback window changes the calculation</h2>



<p class="wp-block-paragraph">Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.</p>



<p class="wp-block-paragraph">What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.</p>



<p class="wp-block-paragraph">It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.</p>



<h2 class="wp-block-heading">Troubleshooting a stalled or failed upgrade</h2>



<p class="wp-block-paragraph">When a node group update fails, the reason comes back as an error code. Start there rather than guessing:</p>



<pre class="wp-block-code"><code>aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup</code></pre>



<ul class="wp-block-list">
<li><strong>PodEvictionFailure.</strong> A budget blocked the drain, or something tolerates all taints. Check <code>ALLOWED DISRUPTIONS</code> first, then tolerations.</li>



<li><strong>NodeCreationFailure in scale up.</strong> New nodes never reached <code>Ready</code>. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.</li>



<li><strong>Slow bootstrap.</strong> A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.</li>



<li><strong>Pods stuck terminating with no budget in sight.</strong> Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.</li>



<li><strong>Insights still in ERROR after the fix.</strong> The thirty day audit window. Refresh, confirm another way, move on.</li>
</ul>



<p class="wp-block-paragraph">One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.</li>



<li>Reaching for <code>OVERWRITE</code> to make a failing add-on update succeed. It succeeds by discarding your configuration.</li>



<li>Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.</li>



<li>Leaving <code>maxUnavailable</code> at its default of one on a large node group, then wondering why the rollout takes hours.</li>



<li>Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.</li>
</ul>



<h2 class="wp-block-heading">How I sequence EKS version upgrades</h2>



<ol class="wp-block-list">
<li>Read the release notes for the target version. Not the summary post, the removals section.</li>



<li>Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.</li>



<li>Audit PDBs and fix every zero. Highest-value ten minutes in the process.</li>



<li>Update add-ons with <code>PRESERVE</code> to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.</li>



<li>Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.</li>



<li>Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.</li>



<li>Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.</li>
</ol>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can you do EKS version upgrades with zero downtime?</h3>



<p class="wp-block-paragraph">For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.</p>



<h3 class="wp-block-heading">Why does the update fail with PodEvictionFailure when my PDBs look fine?</h3>



<p class="wp-block-paragraph">Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check <code>ALLOWED DISRUPTIONS</code> across all namespaces, then check tolerations.</p>



<h3 class="wp-block-heading">Is the force update option safe?</h3>



<p class="wp-block-paragraph">Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.</p>



<h3 class="wp-block-heading">Does Karpenter make upgrades easier than managed node groups?</h3>



<p class="wp-block-paragraph">More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with <code>kubectl</code> rather than through its configuration values, a chart pinned before an API was removed.</p>



<p class="wp-block-paragraph">Which means the pre-flight checks are the upgrade. Ten minutes reading <code>ALLOWED DISRUPTIONS</code> and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your upgrade path?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:</p>



<ul class="wp-block-list">
<li>Audit every PDB against its replica count and flag the ones that will deadlock a drain.</li>



<li>Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.</li>



<li>Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.</li>



<li>Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.</li>



<li>Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.</li>



<li>Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.</li>
</ul>



<p class="wp-block-paragraph">If you are staring at a failed node group update right now, send me the output of <code>kubectl get pdb --all-namespaces</code> and the error code from <code>describe-nodegroup</code>. That is usually enough to say what is holding the drain.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/eks-version-upgrades-without-downtime/">EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</title>
		<link>https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 23 Sep 2026 06:44:01 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost and Usage Report]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logs Insights]]></category>
		<category><![CDATA[Metric Filters]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Runaway Costs]]></category>
		<category><![CDATA[Showback]]></category>
		<category><![CDATA[VPC Flow Logs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=785</guid>

					<description><![CDATA[<p>CloudWatch rarely fails loudly, it accumulates. This is a breakdown of the four independent meters behind the bill: ingestion, storage, Logs Insights scans and custom metric cardinality. Why the same log line gets billed three times, why log groups never expire by default, how a single dimension can multiply your metrics bill without your traffic changing, and how to attribute the spend to specific log groups before you start deleting things.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in finance forwards the AWS invoice with a single line highlighted. AmazonCloudWatch, sitting uncomfortably close to your compute spend. Nobody launched anything. Nobody enabled a new feature. The number just kept climbing, in a slope gentle enough that nobody noticed until it was big enough to be somebody&#8217;s problem.</p>



<p class="wp-block-paragraph">That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.</p>



<p class="wp-block-paragraph">This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.</p>



<h2 class="wp-block-heading">The four meters behind your CloudWatch bill</h2>



<p class="wp-block-paragraph">CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.</p>



<ul class="wp-block-list">
<li><strong>Ingestion.</strong> Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.</li>

<li><strong>Storage.</strong> Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.</li>

<li><strong>Analysis.</strong> Logs Insights bills on data scanned by a query, not on rows returned and not per query.</li>

<li><strong>Metrics and API calls.</strong> Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.</li>
</ul>



<p class="wp-block-paragraph">The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.</p>



<h2 class="wp-block-heading">Ingestion is the meter that actually hurts</h2>



<h3 class="wp-block-heading">Uncompressed in, compressed out</h3>



<p class="wp-block-paragraph">Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.</p>



<p class="wp-block-paragraph">That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The <code>storedBytes</code> value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.</p>



<h3 class="wp-block-heading">The log groups you never created</h3>



<p class="wp-block-paragraph">Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.</p>



<p class="wp-block-paragraph">None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.</p>



<p class="wp-block-paragraph">The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.</p>



<h2 class="wp-block-heading">Retention: the one-line fix almost nobody applies</h2>



<p class="wp-block-paragraph">Log groups default to <strong>Never Expire</strong>. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.</p>



<p class="wp-block-paragraph">Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The <code>?!retentionInDays</code> filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit &#8220;never&#8221; value to match on.</p>



<pre class="wp-block-code"><code>aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Three things are worth knowing before running that across an account.</p>



<ol class="wp-block-list">
<li><strong>The value is an enum, not a free integer.</strong> Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.</li>

<li><strong>Deletion is not instant.</strong> Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported <code>storedBytes</code>, so the billing effect lands before the data physically goes.</li>

<li><strong>Reverting means removing the policy, not setting zero.</strong> To go back to indefinite retention you call <code>delete-retention-policy</code> rather than passing a zero value.</li>
</ol>



<p class="wp-block-paragraph">Dry run before you bulk-apply. This prints what would change without touching anything:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done</code></pre>



<p class="wp-block-paragraph">Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.</p>



<h2 class="wp-block-heading">Cardinality: the multiplier nobody sees coming</h2>



<p class="wp-block-paragraph">This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.</p>



<p class="wp-block-paragraph">CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish <code>OrderLatency</code> with a <code>Region</code> dimension holding three values and you have three metrics. Add a <code>CustomerId</code> dimension and you have three times however many customers you have.</p>



<p class="wp-block-paragraph">The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.</p>



<h3 class="wp-block-heading">Metric filters have a guard rail, and it bites</h3>



<p class="wp-block-paragraph">Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.</p>



<p class="wp-block-paragraph">That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.</p>



<p class="wp-block-paragraph">Audit what exists before adding more, and count what a namespace actually produces:</p>



<pre class="wp-block-code"><code>aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'</code></pre>



<p class="wp-block-paragraph">If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.</p>



<h3 class="wp-block-heading">EMF does exactly what you tell it</h3>



<p class="wp-block-paragraph">Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.</p>



<p class="wp-block-paragraph">The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.</p>



<h3 class="wp-block-heading">Anomaly detection alarms cost more than they look</h3>



<p class="wp-block-paragraph">An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.</p>



<p class="wp-block-paragraph">It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it &#8220;to be safe&#8221; triples a line item for very little detection benefit.</p>



<h2 class="wp-block-heading">Query cost spikes exactly when you need it most</h2>



<p class="wp-block-paragraph">Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.</p>



<p class="wp-block-paragraph">During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:</p>



<ul class="wp-block-list">
<li><strong>Narrow the time window first, then the filter.</strong> Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.</li>

<li><strong>Select specific log groups, not patterns.</strong> Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.</li>
</ul>



<p class="wp-block-paragraph">Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.</p>



<h2 class="wp-block-heading">Finding out where your CloudWatch costs come from</h2>



<p class="wp-block-paragraph">Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.</p>



<ol class="wp-block-list">
<li><strong>Split the bill by meter.</strong> In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: <code>DataProcessing-Bytes</code> is log ingestion, <code>TimedStorage-ByteHrs</code> is log storage, <code>DataScanned-Bytes</code> is Logs Insights, and <code>CW:Requests</code> and <code>CW:GMD-Metrics</code> cover API calls.</li>

<li><strong>Rank log groups by ingestion.</strong> The <code>IncomingBytes</code> metric in the <code>AWS/Logs</code> namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an <em>absolute</em> range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.</li>

<li><strong>Check stored bytes separately.</strong> A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.</li>

<li><strong>Count custom metrics per namespace.</strong> Compare the count against your mental model of what you measure. The gap is cardinality.</li>

<li><strong>Look inside the noisiest group.</strong> Once you know which log group dominates, find out which streams are responsible before changing any application code.</li>
</ol>



<p class="wp-block-paragraph">For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:</p>



<pre class="wp-block-code"><code>stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20</code></pre>



<p class="wp-block-paragraph">That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.</p>



<p class="wp-block-paragraph">If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.</p>



<h2 class="wp-block-heading">Cheaper destinations, and what they cost you</h2>



<p class="wp-block-paragraph">Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.</p>



<ul class="wp-block-list">
<li><strong>The Infrequent Access log class.</strong> Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.</li>

<li><strong>Subscription filter to S3 via Amazon Data Firehose.</strong> Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.</li>

<li><strong>A separate log stack.</strong> Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.</li>
</ul>



<p class="wp-block-paragraph">The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.</p>



<h2 class="wp-block-heading">Troubleshooting a bill that will not go down</h2>



<p class="wp-block-paragraph"><strong>You set retention everywhere and storage barely moved.</strong> Check whether storage was ever the dominant cost. If your usage type breakdown was mostly <code>DataProcessing-Bytes</code>, retention was never going to fix it.</p>



<p class="wp-block-paragraph"><strong>Custom metric count grows with no code changes.</strong> Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.</p>



<p class="wp-block-paragraph"><strong>An alarm went quiet and the bill jumped the same week.</strong> Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm&#8217;s missing data treatment. Alarms treating missing data as not breaching fail silently by design.</p>



<p class="wp-block-paragraph"><strong>API request charges you cannot place.</strong> External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.</p>



<p class="wp-block-paragraph"><strong>Costs dropped, then came back.</strong> Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.</li>

<li>Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.</li>

<li>Filtering logs after they reach CloudWatch. The ingestion charge already applied.</li>

<li>Deleting log groups to cut cost without checking whether anything alarms on them.</li>

<li>Setting retention in the console while the IaC that owns the log group has none configured.</li>

<li>Enabling every EKS control plane log type when only the audit log is consumed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for controlling CloudWatch costs</h2>



<ul class="wp-block-list">
<li><strong>Make retention a required argument.</strong> If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.</li>

<li><strong>Decide verbosity per environment, not per service.</strong> Debug in staging, info in production, with a documented way to raise it temporarily during an incident.</li>

<li><strong>Keep dimensions bounded, identifiers in the message body.</strong> If you cannot enumerate a dimension&#8217;s possible values on a whiteboard, it does not belong in the dimension set.</li>

<li><strong>Alarm on the bill itself.</strong> A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.</li>

<li><strong>Review the top five log groups by ingestion monthly.</strong> Five minutes with the <code>IncomingBytes</code> graph catches new noise before it compounds.</li>

<li><strong>Route archival logs out deliberately.</strong> Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why is my CloudWatch bill so high when my traffic has not changed?</h3>



<p class="wp-block-paragraph">Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.</p>



<h3 class="wp-block-heading">Does deleting a log group reduce my bill immediately?</h3>



<p class="wp-block-paragraph">It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.</p>



<h3 class="wp-block-heading">What counts as one custom metric for billing?</h3>



<p class="wp-block-paragraph">One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.</p>



<h3 class="wp-block-heading">Should I switch log groups to the Infrequent Access class?</h3>



<p class="wp-block-paragraph">For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.</p>



<h3 class="wp-block-heading">Is Logs Insights charged per query or per gigabyte?</h3>



<p class="wp-block-paragraph">Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.</p>



<h3 class="wp-block-heading">Do VPC Flow Logs have to go to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.</p>



<p class="wp-block-paragraph">So attribute first. Split by usage type, rank log groups by <code>IncomingBytes</code>, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.</p>



<h2 class="wp-block-heading">Need a hand with your CloudWatch spend?</h2>



<p class="wp-block-paragraph">This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:</p>



<ul class="wp-block-list">
<li>Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes</li>

<li>Auditing custom metric cardinality and identifying which dimensions are multiplying your series count</li>

<li>Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data</li>

<li>Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment</li>

<li>Designing a subscription filter and S3 archive path for logs you need to keep but rarely query</li>

<li>Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of <code>describe-log-groups</code> for the account. That is usually enough to say where the money is going before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT</title>
		<link>https://john-nessime.com/blog/devops/opentelemetry-on-aws-adot/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 01 Sep 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[ADOT]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon EKS]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[OpenTelemetry]]></category>
		<category><![CDATA[OTLP]]></category>
		<category><![CDATA[Semantic Conventions]]></category>
		<category><![CDATA[SigV4]]></category>
		<category><![CDATA[Tail Sampling]]></category>
		<category><![CDATA[Tracing]]></category>
		<category><![CDATA[Transaction Search]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[X-Ray]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=523</guid>

					<description><![CDATA[<p>A pipeline that returns 200 is not a pipeline that works. The four decisions behind running OpenTelemetry on AWS with ADOT, the CloudWatch OTLP endpoint limits that drop data without erroring, and how to catch the failures before a colleague does.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/opentelemetry-on-aws-adot/">Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A colleague messages you: &#8220;checkout fell off the service map.&#8221; The service is fine. It&#8217;s serving traffic, the collector pod is <code>Running</code>, the exporter queue is empty, and nothing in the collector log reads like an error. But a chunk of the spans that left the application never showed up in CloudWatch.</p>



<p class="wp-block-paragraph">That&#8217;s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered <code>200</code>. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.</p>



<p class="wp-block-paragraph">Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.</p>



<h2 class="wp-block-heading">What ADOT actually buys you</h2>



<p class="wp-block-paragraph">ADOT is not a separate protocol or agent. It&#8217;s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.</p>



<p class="wp-block-paragraph">So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn&#8217;t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.</p>



<h2 class="wp-block-heading">Decision one: where the collector runs</h2>



<p class="wp-block-paragraph">Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.</p>



<h3 class="wp-block-heading">Sidecar</h3>



<p class="wp-block-paragraph">One collector container per ECS task or pod. The application talks to <code>localhost</code>, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.</p>



<h3 class="wp-block-heading">Agent plus gateway</h3>



<p class="wp-block-paragraph">A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.</p>



<p class="wp-block-paragraph">The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.</p>



<h3 class="wp-block-heading">No collector at all</h3>



<p class="wp-block-paragraph">The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.</p>



<h2 class="wp-block-heading">Decision two: which door into AWS</h2>



<p class="wp-block-paragraph">Two ways to hand telemetry to AWS, and they behave very differently.</p>



<p class="wp-block-paragraph">The older path uses AWS-specific exporters. <code>awsxray</code> converts OTLP spans into X-Ray segment documents and calls the X-Ray API. <code>awsemf</code> converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.</p>



<p class="wp-block-paragraph">The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain <code>otlphttp</code> exporter and a SigV4 signer. Same exporter you&#8217;d point at Grafana Cloud or Honeycomb, different URL and authenticator. That&#8217;s what &#8220;export anywhere&#8221; means in practice, and it&#8217;s why I default to it on new work.</p>



<p class="wp-block-paragraph">The endpoints follow a per-service pattern:</p>



<ul class="wp-block-list">
<li>Traces: <code>https://xray.&lt;region&gt;.amazonaws.com/v1/traces</code></li>

<li>Metrics: <code>https://monitoring.&lt;region&gt;.amazonaws.com/v1/metrics</code></li>

<li>Logs: <code>https://logs.&lt;region&gt;.amazonaws.com/v1/logs</code></li>
</ul>



<p class="wp-block-paragraph">Three constraints there will each cost you an afternoon if you don&#8217;t know them going in.</p>



<p class="wp-block-paragraph"><strong>HTTP only.</strong> No gRPC. If your collector exports over <code>otlp</code> on 4317 you can&#8217;t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it&#8217;s the outbound leg that must be HTTP.</p>



<p class="wp-block-paragraph"><strong>SigV4 required.</strong> That means the <code>sigv4auth</code> extension, configured per signal because the signing service name differs: <code>xray</code>, <code>monitoring</code>, and <code>logs</code> respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.</p>



<p class="wp-block-paragraph"><strong>Logs need headers, not just a URL.</strong> Target log group and stream travel in <code>x-aws-log-group</code> and <code>x-aws-log-stream</code> headers. Omit them and the request has nowhere to land.</p>



<p class="wp-block-paragraph">A working shape for traces and logs looks like this:</p>



<pre class="wp-block-code"><code>extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]</code></pre>



<p class="wp-block-paragraph">Note <code>traces_endpoint</code> and <code>logs_endpoint</code> rather than <code>endpoint</code>. This one bites people. The <code>endpoint</code> setting is a <em>base URL</em> and the exporter appends the signal path itself, so <code>endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics</code> actually requests <code>/v1/metrics/v1/metrics</code>. The per-signal settings take a full path. If you&#8217;re getting 404s from a URL you&#8217;re certain is right, this is why.</p>



<p class="wp-block-paragraph">Also worth a comment in your config: upstream has renamed the component to <code>otlp_http</code> and <code>otlphttp</code> is now a deprecated alias scheduled for removal. Both work today.</p>



<h3 class="wp-block-heading">Traces need Transaction Search turned on first</h3>



<p class="wp-block-paragraph">This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:</p>



<pre class="wp-block-code"><code>aws xray update-trace-segment-destination --destination CloudWatchLogs</code></pre>



<p class="wp-block-paragraph">Spans then land in a log group named <code>aws/spans</code>, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs <code>xray:UpdateTraceSegmentDestination</code> and <code>xray:UpdateIndexingRule</code> plus log group creation rights.</p>



<p class="wp-block-paragraph">Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it&#8217;s a line item rather than a rounding error. And AWS recommends <code>always_on</code> sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.</p>



<h3 class="wp-block-heading">The X-Ray SDK clock is running</h3>



<p class="wp-block-paragraph">The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray <em>service</em> is fine and still gaining features. It&#8217;s the client libraries and the UDP daemon winding down. If you run <code>aws-xray-sdk</code> and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.</p>



<h2 class="wp-block-heading">Decision three: metrics, temporality, and the cardinality bill</h2>



<p class="wp-block-paragraph">Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.</p>



<p class="wp-block-paragraph">The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor&#8217;s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back <code>200</code> with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS&#8217;s own examples use a conservative value:</p>



<pre class="wp-block-code"><code>processors:
  batch:
    send_batch_size: 200
    timeout: 10s</code></pre>



<p class="wp-block-paragraph">The other limits shape attribute design more than config:</p>



<ul class="wp-block-list">
<li>150 labels maximum across resource, scope, and datapoint attributes per datapoint</li>

<li>40 KB combined label and value size per series per datapoint</li>

<li>One million new series creatable per ten-minute window, per account</li>

<li>Timestamps no more than ten minutes in the future or fourteen days in the past</li>
</ul>



<p class="wp-block-paragraph">That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You&#8217;ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.</p>



<p class="wp-block-paragraph">Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch&#8217;s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The <code>cumulativetodelta</code> processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.</p>



<p class="wp-block-paragraph">Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.</p>



<h2 class="wp-block-heading">Decision four: a second backend without a second instrumentation</h2>



<p class="wp-block-paragraph">Here&#8217;s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:</p>



<pre class="wp-block-code"><code>service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]</code></pre>



<p class="wp-block-paragraph">The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That&#8217;s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.</p>



<p class="wp-block-paragraph">Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.</p>



<p class="wp-block-paragraph">It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.</p>



<h2 class="wp-block-heading">Why OpenTelemetry on AWS fails quietly</h2>



<p class="wp-block-paragraph">Most breakages here don&#8217;t throw. How to recognise the common ones:</p>



<h3 class="wp-block-heading">Data is missing but nothing errors</h3>



<p class="wp-block-paragraph">Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector&#8217;s internal telemetry and compare the exporter&#8217;s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.</p>



<h3 class="wp-block-heading">403 with a signature mismatch</h3>



<p class="wp-block-paragraph">A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since <code>xray</code>, <code>monitoring</code>, and <code>logs</code> are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.</p>



<h3 class="wp-block-heading">Backfilled data vanishes</h3>



<p class="wp-block-paragraph">Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.</p>



<h3 class="wp-block-heading">Traces arrive but the service map is empty</h3>



<p class="wp-block-paragraph">Usually a missing or inconsistent <code>service.name</code> resource attribute, or double sampling between the SDK and the indexing rule. Confirm what&#8217;s actually leaving the collector before hunting in the console: add the <code>debug</code> exporter to a copy of the pipeline, set <code>verbosity: detailed</code>, and read the resource attributes on real spans.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They&#8217;re HTTP only, and the error won&#8217;t say so clearly.</li>

<li>Using <code>endpoint</code> with a full signal path instead of the per-signal <code>traces_endpoint</code>, <code>metrics_endpoint</code>, or <code>logs_endpoint</code>.</li>

<li>Leaving the batch processor at its default size on the metrics pipeline.</li>

<li>Sampling in the SDK <em>and</em> relying on Transaction Search indexing rules, halving visibility twice over.</li>

<li>Putting request IDs, user IDs, or raw paths into metric attributes.</li>

<li>Running a gateway pool with no alerting on the collector&#8217;s own health, so saturation is invisible until you need it.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.</li>

<li>Set <code>service.name</code>, <code>service.version</code>, and <code>deployment.environment</code> as resource attributes everywhere. Almost every correlation feature depends on them.</li>

<li>Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn&#8217;t depend on the pipeline being healthy.</li>

<li>Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.</li>

<li>Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.</li>

<li>Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn&#8217;t work.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need a collector if CloudWatch accepts OTLP directly?</h3>



<p class="wp-block-paragraph">Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.</p>



<h3 class="wp-block-heading">Can I send OTLP to CloudWatch over gRPC?</h3>



<p class="wp-block-paragraph">No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can&#8217;t forward over it.</p>



<h3 class="wp-block-heading">What&#8217;s the difference between ADOT and the upstream collector?</h3>



<p class="wp-block-paragraph">Same codebase, different build. ADOT is AWS&#8217;s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it&#8217;s a support and packaging choice, not an architectural one.</p>



<h3 class="wp-block-heading">Do I have to enable Transaction Search to send traces?</h3>



<p class="wp-block-paragraph">For the X-Ray OTLP endpoint, yes. It&#8217;s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the <code>awsxray</code> exporter against the classic X-Ray API instead doesn&#8217;t require it, but you give up the span-level analytics Transaction Search provides.</p>



<h3 class="wp-block-heading">Can I ship telemetry to CloudWatch from outside AWS?</h3>



<p class="wp-block-paragraph">For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.</p>



<h3 class="wp-block-heading">Is it urgent to migrate off the X-Ray SDK?</h3>



<p class="wp-block-paragraph">Not an emergency, but it&#8217;s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won&#8217;t get new library instrumentation. Plan the migration on your schedule rather than someone else&#8217;s.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A pipeline that returns <code>200</code> is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?</p>



<p class="wp-block-paragraph">Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and &#8220;instrument once, export anywhere&#8221; stops being a slogan and becomes a property you can test.</p>



<h2 class="wp-block-heading">Need help with your OpenTelemetry pipeline on AWS?</h2>



<p class="wp-block-paragraph">Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches</li>

<li>Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling</li>

<li>Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support</li>

<li>Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning</li>

<li>Dual export to CloudWatch and a third-party backend, so a future switch is a config change</li>

<li>Meta-monitoring that tells you the pipeline is broken before your users do</li>
</ul>



<p class="wp-block-paragraph">Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/opentelemetry-on-aws-adot/">Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</title>
		<link>https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Adaptive Bitrate]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[MediaConvert]]></category>
		<category><![CDATA[Normalized Minutes]]></category>
		<category><![CDATA[QVBR]]></category>
		<category><![CDATA[Reserved Transcode Slots]]></category>
		<category><![CDATA[Video Transcoding]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=542</guid>

					<description><![CDATA[<p>One checkbox can reprice every output in a MediaConvert job. A working breakdown of how the per-minute charge is assembled from normalized minutes and per-output billing, why failed jobs are free but still expensive, and how to build retry and recovery that classifies before it retries.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A turnaround complaint lands: the promo team&#8217;s masters take too long to appear in the CMS. Someone opens the job template, finds a setting called accelerated transcoding, ticks it, and moves on. Turnaround improves. Nobody mentions it again.</p>



<p class="wp-block-paragraph">The invoice is where it surfaces, and not only on the big outputs. The 240p ladder rung went up. The audio-only rendition went up. Every output in every job that template touches went up, including the ones that never needed the feature.</p>



<p class="wp-block-paragraph">That is not a billing bug. It is the documented rule working as designed, and it is the most common reason a bill moves without anyone changing how much video goes through it. Understanding <strong>MediaConvert cost per minute</strong> starts with accepting that the unit you pay for is not a job, not a source file, and not a minute of compute.</p>



<p class="wp-block-paragraph">What follows is how that charge is assembled, what happens to it when a job fails, and how to build recovery around AWS Elemental MediaConvert. Rates are deliberately absent; AWS changes them and they vary by region. The mechanism is what catches people.</p>



<h2 class="wp-block-heading">How MediaConvert cost per minute is actually assembled</h2>



<p class="wp-block-paragraph">Three rules stack on top of each other. Miss one and your cost model is wrong by a multiple, not a rounding error.</p>



<h3 class="wp-block-heading">You pay per output, not per job</h3>



<p class="wp-block-paragraph">An eight-rung ABR ladder plus an audio-only rendition is nine billable items, each charged for its own duration. AWS bills the duration of the finished assets, not the transcoding time, in hundredth-of-a-minute increments. A ten-minute source producing nine outputs bills as ninety minutes of output.</p>



<p class="wp-block-paragraph">Ladder design is therefore a cost decision, not only a quality one. The bottom rungs are not cheap because they are small. They cost a full pass of the source duration at whatever multiplier their settings attract, whether anyone streams them or not.</p>



<h3 class="wp-block-heading">Normalized minutes, and why multipliers compound</h3>



<p class="wp-block-paragraph">There is no flat rate per output minute. MediaConvert converts each output into normalized minutes, a unit that scales with the work the encode demanded. Resolution, frame rate, codec and number of encoding passes each carry a multiplier, and those multipliers combine. Single-pass to multi-pass is one. 30fps to 60fps is another. Do both to a UHD rung and you have stacked three.</p>



<p class="wp-block-paragraph">Volume discounts then apply to your accumulated normalized minutes for the month, per region, with basic-tier and professional-tier usage accruing separately toward their own thresholds. Two consequences: splitting a workload across regions splits the accrual and can leave you paying a higher effective rate in both, and under consolidated billing in AWS Organizations usage accrues across accounts, so a per-account model understates the discount you actually get.</p>



<h3 class="wp-block-heading">The job-wide setting that reprices everything</h3>



<p class="wp-block-paragraph">Back to the opening. Each output is assessed for its billing tier separately, but only when the job has no job-wide professional-tier feature enabled. Turn one on, accelerated transcoding being the usual culprit, and every output in that job bills at pro-tier rates. An SD rung that would have billed at basic tier gets repriced because a different output in the same job needed to finish faster.</p>



<p class="wp-block-paragraph">If a subset of content genuinely needs acceleration, split it out. Long, visually complex assets get a template with the feature; routine web renditions get one without. Two templates is less tidy than one, and considerably cheaper than a single checkbox repricing your catalogue.</p>



<h2 class="wp-block-heading">Failed jobs are free, which changes how you should retry</h2>



<p class="wp-block-paragraph">MediaConvert charges only when a job reaches COMPLETED. Jobs ending in ERROR are not billed. That sounds like it removes the urgency from failure handling. It does the opposite.</p>



<p class="wp-block-paragraph">A job that fails on output seven of nine may still have written outputs one through six to S3. Those partial artefacts cost storage, and if a naive retry writes to the same prefix you get a mix of old and new segments that a player will serve quite happily until someone notices the ladder is inconsistent. Free at the transcode line, expensive everywhere downstream.</p>



<p class="wp-block-paragraph">&#8220;Not billed&#8221; also makes blind retries feel harmless, so people write them. A loop against a deterministic failure, a corrupt source or a settings mismatch, will never succeed. It burns concurrency paying jobs need and produces identical alerts until the team ignores the channel. The service already retries some conditions itself and exposes a retry count, so read that field before adding an attempt of your own.</p>



<h2 class="wp-block-heading">The queue model is a throughput control, not a cost control</h2>



<p class="wp-block-paragraph">On-demand queues bill per output minute. How many jobs run at once is governed by a service quota for concurrent jobs per account in the region, a quota per individual queue, and a concurrent-jobs setting on each queue. Creating a second queue does not conjure capacity out of the account; you raise it through Service Quotas. Jobs sitting in SUBMITTED rather than moving to PROGRESSING is the signal you have hit the ceiling.</p>



<p class="wp-block-paragraph">Reserved queues work on a different axis. You buy reserved transcode slots against a twelve-month commitment and pay for that capacity regardless of use. Each slot has its own dedicated compute, so one slot runs a given job just as fast as ten would. More slots means more jobs in parallel, not faster jobs. Three things about the commitment deserve a second read before anyone clicks agree:</p>



<ul class="wp-block-list">
<li>You cannot cancel it, and you cannot decrease the slot count.</li>

<li>Adding capacity does not extend the existing term. It starts a fresh twelve-month commitment for the larger count, from the day you add it.</li>

<li>When the plan expires, the queue persists and keeps accepting jobs. MediaConvert just does not run them.</li>
</ul>



<p class="wp-block-paragraph">That last one is the invisible failure in this whole topic. Your submission code gets a job ID back. The API call succeeded. Nothing errors. The jobs simply never progress, and unless you alarm on queue standby time or on the gap between jobs submitted and completed, the first person to notice is whoever asks why the library stopped updating. Set auto-renew deliberately, either way, and diary the term end date.</p>



<p class="wp-block-paragraph">Queue hopping is the pressure valve. Give a job a wait time and a destination queue, and if it is still waiting when the timer expires it moves. The usual shape is reserved to on-demand during a spike. Billing follows the queue the job actually runs in, so a hopped job becomes on-demand spend.</p>



<pre class="wp-block-code"><code>{
  "Settings": { "OutputGroups": [], "Inputs": [] },
  "HopDestinations": [
    {
      "WaitMinutes": 10,
      "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/ondemandqueue",
      "Priority": 25
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">WaitMinutes accepts 1 to 4320. Priority runs from -50 to 50 and defaults to 0, and the optional priority under HopDestinations lets a hopped job land higher or lower in the destination queue than it sat in the original. Hopping is a cost decision as much as a throughput one: it converts capacity you have already paid for into incremental on-demand spend.</p>



<h2 class="wp-block-heading">The failure families that actually bite</h2>



<p class="wp-block-paragraph">MediaConvert publishes numbered error codes with documented causes. Rather than reproduce the table, group them by what you have to do about them, because the correct response differs sharply between groups.</p>



<h3 class="wp-block-heading">Access failures</h3>



<p class="wp-block-paragraph">The service could not open an input or output location. Causes cluster around IAM permissions on the role the job runs under, cross-account bucket policies when the source lives elsewhere, keys that do not exist, and files whose headers cannot be read. Retry these once with a delay, because uploads still in flight are a real condition. If the second attempt fails identically, it is configuration, and no amount of retrying fixes configuration.</p>



<h3 class="wp-block-heading">Settings failures that only appear on some inputs</h3>



<p class="wp-block-paragraph">The nastiest to diagnose, because the template has been fine for the last thousand assets. An encoder initialisation failure on one rung, triggered by an unusual source aspect ratio or frame rate meeting a fixed constraint in the template, is the classic shape. Never retry these; the job fails identically forever. The fix is usually to loosen the constraint rather than pin it harder, because letting the service derive a value the source cannot satisfy removes the whole class of failure.</p>



<h3 class="wp-block-heading">Warnings, which are not errors and are easy to miss</h3>



<p class="wp-block-paragraph">MediaConvert also returns warning codes for problems that do not stop the job. It completes, you get billed, and something about the output is wrong. Video padding inserted to keep audio and video aligned, black frames carried through from the source, a QVBR score well below the rest of the catalogue: none raise an error, all are visible in metrics. Alert only on ERROR and this entire family is invisible to you.</p>



<h3 class="wp-block-heading">Pinned engine versions that quietly expire</h3>



<p class="wp-block-paragraph">MediaConvert groups its releases into job engine versions so you can hold production on one while validating another. Access is requested through AWS Support rather than open to every account, so check whether it applies to you first. The behaviour worth knowing is expiry: requesting an expired version does not fail. The job runs as though no version had been specified, and the field reporting the version used comes back empty. An invalid version, by contrast, returns a 400 and creates no job. The failure mode of pinning is silence.</p>



<h2 class="wp-block-heading">Building failure recovery that does not cost you twice</h2>



<p class="wp-block-paragraph">The pattern that holds up is event-driven, classified and idempotent. In order:</p>



<ol class="wp-block-list">
<li>Subscribe to job state changes in EventBridge rather than polling. The ERROR event carries the code and message in its detail block.</li>

<li>Route by error code, not by status: one target for retryable access conditions, another for deterministic settings failures that need a human, and a suppression path for anything already seen on that asset.</li>

<li>Write a retry&#8217;s outputs to a new prefix, or clear the old one first. Never let attempt two land beside attempt one&#8217;s partial results.</li>

<li>Send a client request token on job creation. Reusing the same token within a minute of a successful request returns the original job instead of creating a duplicate, which protects you when your orchestration retries a create call whose response it never saw.</li>

<li>Cap attempts and land the remainder in a dead letter queue with the original event attached. A failure nobody can see is a failure nobody fixes.</li>

<li>Carry your own identifiers through user metadata on the job, so the returning event joins back to the asset without a lookup table.</li>
</ol>



<pre class="wp-block-code"><code>{
  "source": ["aws.mediaconvert"],
  "detail-type": ["MediaConvert Job State Change"],
  "detail": {
    "status": ["ERROR"]
  }
}</code></pre>



<p class="wp-block-paragraph">The matching event delivers a detail object containing the queue ARN, job ID, status, an errorCode and an errorMessage. Route on errorCode. That one decision is the difference between a recovery system and a retry loop.</p>



<h2 class="wp-block-heading">Measuring what you spend before the invoice tells you</h2>



<p class="wp-block-paragraph">MediaConvert sends metrics to CloudWatch at the end of every job, several mapping directly onto the billing dimensions. The ones I reach for first:</p>



<ul class="wp-block-list">
<li><code>SDOutputDuration</code>, <code>HDOutputDuration</code>, <code>UHDOutputDuration</code>, <code>8KOutputDuration</code> and <code>AudioOutputDuration</code>, in milliseconds per queue. Summed over a month, that is billable output volume by resolution class, days ahead of Cost Explorer.</li>

<li><code>JobErrorsByCode</code>, counting jobs that failed with a particular code. Build the dashboard on this one; it shows which failure family is growing.</li>

<li><code>StandbyTime</code>, how long a job waits before processing starts. Alarm on it and you catch backlogs, and you catch an expired reserved plan.</li>

<li><code>JobsSubmittedCount</code> against <code>JobsCompletedCount</code>. A widening gap is the earliest honest signal something upstream is broken.</li>

<li><code>QVBRAvgQualityLowBitrate</code> and <code>QVBRMinQualityHighBitrate</code>, for quality regressions that never raise an error.</li>
</ul>



<p class="wp-block-paragraph">For attribution rather than volume, enable cost allocation tags and set the billing tags source on your jobs so per-output costs land against a tag you chose. Pushing that into a wider FinOps view through Vantage, CloudZero or a Grafana Cloud dashboard fed from CloudWatch gets you per-title cost without building the aggregation yourself. Either way, these metrics let you model MediaConvert cost per minute continuously instead of discovering it monthly.</p>



<p class="wp-block-paragraph">Transcoding is only part of the line. Storage for sources and renditions in S3, and egress through Amazon CloudFront or whichever CDN fronts delivery, are separate charges and often the larger ones. A cost model that stops at the encoder will be wrong.</p>



<h2 class="wp-block-heading">Troubleshooting a failed job</h2>



<ol class="wp-block-list">
<li>Get the error code first. Everything else follows from which family it belongs to.</li>

<li>Ask whether other assets in the same bucket succeeded in the same window. Yes points at the file, no points at permissions or configuration.</li>

<li>For a settings failure, bisect the job. Strip it to a single video output, confirm it runs, then add outputs and features back one at a time. Tedious, and faster than reasoning about it.</li>

<li>Check warnings on jobs that completed nearby. Warnings often precede errors from the same root cause.</li>

<li>If nothing ran at all, look at the queue before the job: plan status on a reserved queue, standby time on an on-demand one.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Modelling cost from source duration instead of total output duration across the ladder.</li>

<li>Enabling a job-wide professional-tier feature in a shared template and repricing every output in it.</li>

<li>Creating extra on-demand queues for throughput when the account quota is the real ceiling.</li>

<li>Buying reserved slots off a peak-week estimate and paying for idle capacity the other fifty-one.</li>

<li>Retrying every failure identically, including the deterministic ones that cannot succeed.</li>

<li>Alerting on ERROR only, and never looking at warnings or quality scores.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Separate templates by cost profile rather than content type. Anything with a job-wide pro-tier feature gets its own.</li>

<li>Audit the ladder against playback data at least once. Rungs nobody streams are pure cost.</li>

<li>Keep a workload in one region so volume discounts accrue against a single pool.</li>

<li>Run a representative job on an on-demand queue with the reserved simulation setting enabled before committing to slots.</li>

<li>Use job priority instead of extra queues when the goal is for urgent work to jump ahead.</li>

<li>Derive client request tokens from the asset, not from a timestamp, so retries are genuinely idempotent.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I pay for MediaConvert jobs that fail?</h3>



<p class="wp-block-paragraph">No. Billing applies only to jobs reaching COMPLETED status, so a job ending in ERROR is not charged. You may still be paying for partial outputs that reached S3 before the failure, and the storage they occupy until something cleans them up.</p>



<h3 class="wp-block-heading">What is a normalized minute?</h3>



<p class="wp-block-paragraph">The billing unit MediaConvert uses so a minute of demanding output counts for more than a minute of simple output. Resolution, frame rate, codec and encoding passes each contribute a multiplier, and those combine into the normalized total your volume discount tier is measured against.</p>



<h3 class="wp-block-heading">Is a reserved queue cheaper than on demand?</h3>



<p class="wp-block-paragraph">Only if utilisation is genuinely steady. You pay for reserved transcode slots regardless of use, across a twelve-month commitment you cannot cancel or shrink. Bursty catalogues usually do better on demand, where volume discounts apply automatically as monthly usage climbs. Reserved pricing also excludes some features, so confirm everything your templates use is available there before committing.</p>



<h3 class="wp-block-heading">Why are my jobs stuck in SUBMITTED?</h3>



<p class="wp-block-paragraph">On an on-demand queue, it is already running its maximum number of concurrent jobs; raise the quota through Service Quotas rather than adding queues. On a reserved queue, check whether the pricing plan has expired, because an expired queue keeps accepting jobs and never processes them.</p>



<h3 class="wp-block-heading">Does adding queues give me more transcoding capacity?</h3>



<p class="wp-block-paragraph">No. Concurrency is bounded by account-level and queue-level quotas in the region. Extra queues isolate workloads from each other, for instance keeping one client&#8217;s jobs from starving another&#8217;s. Capacity increases come from Service Quotas.</p>



<h3 class="wp-block-heading">How do I stop one setting from repricing my whole job?</h3>



<p class="wp-block-paragraph">Split the job. Job-wide professional-tier features apply their tier to every output, so isolate the assets that need them into their own template and keep routine renditions in a template with no job-wide pro-tier features enabled.</p>



<h3 class="wp-block-heading">Should I run FFmpeg on my own servers instead?</h3>



<p class="wp-block-paragraph">Sometimes. Predictable, modest volume on dedicated hardware from a provider like Contabo or InterServer can undercut per-minute pricing, and you get exact control over encoder flags. What you take on is capacity planning, codec licensing questions, and building the retry and observability machinery the managed service hands you for free. That trade favours MediaConvert for spiky workloads and broadcast-grade features, and self-hosting for steady, simple ladders where an engineer already owns the pipeline.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">MediaConvert cost per minute is not a rate you look up. It is a product: output count, times duration, times a stack of feature multipliers, with a single job-wide setting able to reprice the lot. Get that model into whatever you plan from and the bill stops being a monthly surprise.</p>



<p class="wp-block-paragraph">On the failure side, the codes are documented and the events are structured, so there is no excuse for a retry loop that cannot tell a permissions problem from a settings problem. Classify before you retry. And put an alarm on queue standby time today, because the failure that costs you most is the one where nothing errors at all.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your transcoding pipeline?</h2>



<p class="wp-block-paragraph">I work with teams running video-on-demand pipelines on AWS, usually at the point where the bill or the failure rate has become someone&#8217;s problem. What I actually help with here:</p>



<ul class="wp-block-list">
<li>Auditing job templates and ABR ladders for outputs and multipliers costing more than they return.</li>

<li>Building a cost model from your CloudWatch output-duration metrics so you can forecast rather than react.</li>

<li>Designing EventBridge-driven failure handling that classifies by error code and retries only what can succeed.</li>

<li>Working the reserved versus on-demand decision against your real utilisation, including whether queue hopping earns its keep.</li>

<li>Debugging settings failures that only hit certain sources, and loosening the constraints causing them.</li>

<li>Wiring transcoding, S3 lifecycle and CDN delivery into one cost and reliability picture instead of three.</li>
</ul>



<p class="wp-block-paragraph">Send me a job template, an error code, or a month of output-duration metrics and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</title>
		<link>https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 26 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon EBS]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[NAT Gateway]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reserved Instances]]></category>
		<category><![CDATA[Savings Plans]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=264</guid>

					<description><![CDATA[<p>Most AWS cost work goes wrong because it starts with the biggest line item, which is also the riskiest. Order the work by blast radius instead: free networking and storage fixes first, performance envelopes one workload at a time, and commitments last. Includes the commands to find the waste and the cuts that look harmless and are not.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/">Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on the third or fourth of the month, and it is never phrased as a question. Someone from finance pastes a screenshot of the AWS bill into a channel, adds a number and a percentage, and waits. Nobody says &#8220;please cause an outage.&#8221; What they say is &#8220;can we get this down.&#8221;</p>



<p class="wp-block-paragraph">That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that &#8220;went fine.&#8221;</p>



<p class="wp-block-paragraph">This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.</p>



<h2 class="wp-block-heading">The cuts that break production are rarely the ones you fear</h2>



<p class="wp-block-paragraph">The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.</p>



<p class="wp-block-paragraph">The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.</p>



<p class="wp-block-paragraph">So the rule I work by: <strong>sequence cost work by blast radius, not by savings size</strong>. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">First, find out what you are actually paying for</h2>



<p class="wp-block-paragraph">The default Cost Explorer view groups by service, which is close to useless here. &#8220;EC2 &#8211; Other&#8221; is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.</p>



<h3 class="wp-block-heading">Group by usage type instead</h3>



<p class="wp-block-paragraph">Usage type names the actual mechanism being billed. Strings like <code>NatGateway-Bytes</code>, <code>NatGateway-Hours</code>, <code>PublicIPv4:InUseAddress</code> and <code>EBS:VolumeUsage.gp2</code> tell you what to go fix. This pulls last month&#8217;s usage types sorted by cost:</p>



<pre class="wp-block-code"><code>aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30</code></pre>



<p class="wp-block-paragraph">That uses GNU <code>date</code>, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.</p>



<h3 class="wp-block-heading">Billing data cannot tell you where your bytes went</h3>



<p class="wp-block-paragraph">This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.</p>



<p class="wp-block-paragraph">VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier zero: savings production cannot notice</h2>



<p class="wp-block-paragraph">Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.</p>



<h3 class="wp-block-heading">The orphan sweep</h3>



<p class="wp-block-paragraph">AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.</p>



<pre class="wp-block-code"><code># Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text</code></pre>



<p class="wp-block-paragraph">Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.</p>



<h3 class="wp-block-heading">NAT Gateway is usually the largest risk-free win</h3>



<p class="wp-block-paragraph">A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.</p>



<p class="wp-block-paragraph">The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.</p>



<pre class="wp-block-code"><code>aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0</code></pre>



<p class="wp-block-paragraph">A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.</p>



<p class="wp-block-paragraph">Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.</p>



<h3 class="wp-block-heading">Log groups that never expire</h3>



<p class="wp-block-paragraph">CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.</p>



<pre class="wp-block-code"><code># Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.</p>



<h3 class="wp-block-heading">Incomplete multipart uploads</h3>



<p class="wp-block-paragraph">Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:</p>



<pre class="wp-block-code"><code>{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Apply it with <code>aws s3api put-bucket-lifecycle-configuration</code>. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier one: savings that move a performance envelope</h2>



<p class="wp-block-paragraph">Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.</p>



<h3 class="wp-block-heading">gp2 to gp3, and the throughput trap</h3>



<p class="wp-block-paragraph">This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.</p>



<p class="wp-block-paragraph">The trap is throughput, not IOPS. gp3&#8217;s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is &#8220;the nightly job got slower&#8221; rather than an alert.</p>



<p class="wp-block-paragraph">Check <code>VolumeReadBytes</code> and <code>VolumeWriteBytes</code> in CloudWatch first, and provision throughput explicitly where the volume needs it:</p>



<pre class="wp-block-code"><code>aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text</code></pre>



<p class="wp-block-paragraph">Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.</p>



<h3 class="wp-block-heading">Rightsizing without guessing</h3>



<p class="wp-block-paragraph">AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.</p>



<pre class="wp-block-code"><code>aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text</code></pre>



<p class="wp-block-paragraph">My working rules:</p>



<ul class="wp-block-list">
<li>Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.</li>

<li>Change one dimension at a time, family or size, so you know what caused any regression.</li>

<li>Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.</li>

<li>Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.</li>

<li>Write down the metric you expect to move and the threshold at which you revert, before you make the change.</li>
</ul>



<h3 class="wp-block-heading">S3 storage classes and the small-object trap</h3>



<p class="wp-block-paragraph">Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.</p>



<p class="wp-block-paragraph">Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.</p>



<p class="wp-block-paragraph">Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier two: commitments, which break budgets rather than uptime</h2>



<p class="wp-block-paragraph">Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:</p>



<ul class="wp-block-list">
<li>Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.</li>

<li>Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.</li>

<li>Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.</li>

<li>Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.</li>

<li>Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.</li>

<li>Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.</li>
</ul>



<p class="wp-block-paragraph">Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.</p>



<p class="wp-block-paragraph">One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill went down and something broke</h2>



<p class="wp-block-paragraph">The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:</p>



<ul class="wp-block-list">
<li><strong>Latency got worse but CPU is fine.</strong> Look at storage. <code>VolumeQueueLength</code> climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling <code>BurstBalance</code> is the tell, and it usually starts hours before anything is user-visible.</li>

<li><strong>Intermittent timeouts after rightsizing.</strong> Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.</li>

<li><strong>Something stopped reaching a service.</strong> If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint&#8217;s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.</li>

<li><strong>Objects are slow or missing from an application path.</strong> Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.</li>
</ul>



<p class="wp-block-paragraph">Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Starting with the biggest number.</strong> Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.</li>

<li><strong>Committing before rightsizing.</strong> This locks in the exact waste you were trying to remove, for years.</li>

<li><strong>Accepting gp3 defaults on high-throughput volumes.</strong> Cheaper storage, halved throughput ceiling, no alert.</li>

<li><strong>Enabling Intelligent-Tiering on buckets full of tiny objects.</strong> Monitoring fees on objects that will never move a tier.</li>

<li><strong>Turning off monitoring to save on monitoring.</strong> Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.</li>

<li><strong>Doing it once.</strong> Cost drifts back within two quarters unless something structural stops it.</li>
</ul>



<h2 class="wp-block-heading">Best practices that make the savings stick</h2>



<ul class="wp-block-list">
<li><strong>Make cost visible before it is a crisis.</strong> AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.</li>

<li><strong>Tag at creation, in the IaC.</strong> Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.</li>

<li><strong>Put non-production on a schedule.</strong> Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.</li>

<li><strong>Make the cheap thing the default in the module.</strong> gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.</li>

<li><strong>Review commitment coverage on a schedule.</strong> Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the fastest way to cut AWS costs without breaking production?</h3>



<p class="wp-block-paragraph">Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.</p>



<h3 class="wp-block-heading">Why is my &#8220;EC2 &#8211; Other&#8221; line item so large?</h3>



<p class="wp-block-paragraph">It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.</p>



<h3 class="wp-block-heading">Is migrating from gp2 to gp3 always cheaper?</h3>



<p class="wp-block-paragraph">No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.</p>



<h3 class="wp-block-heading">Can I cancel a Savings Plan if my usage drops?</h3>



<p class="wp-block-paragraph">Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.</p>



<h3 class="wp-block-heading">Should I use Spot Instances to cut costs?</h3>



<p class="wp-block-paragraph">For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.</p>



<h3 class="wp-block-heading">How do I find out where my NAT Gateway traffic is going?</h3>



<p class="wp-block-paragraph">Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.</p>



<p class="wp-block-paragraph">Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say &#8220;not yet&#8221; when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your AWS bill?</h2>



<p class="wp-block-paragraph">This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:</p>



<ul class="wp-block-list">
<li>A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.</li>

<li>NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.</li>

<li>EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.</li>

<li>CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.</li>

<li>Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.</li>

<li>Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/">Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</title>
		<link>https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 26 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alarm Fatigue]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Composite Alarms]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[On-Call]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=266</guid>

					<description><![CDATA[<p>Most AWS accounts have dozens of CloudWatch alarms and about three anyone trusts. A practical guide to missing-data settings, M-out-of-N evaluation, metric math, composite alarms and mute rules, and how to test the notification path before you need it.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/">CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The CPU alarm has been green for six weeks. That is not because the instance is healthy. It is because the agent stopped publishing the metric, the alarm quietly slid into INSUFFICIENT_DATA, and nobody ever wired that state to anything. Green on the dashboard, deaf in production.</p>



<p class="wp-block-paragraph">That is the failure mode nobody catches during a review, and it is why most AWS accounts have dozens of alarms and about three anyone trusts. The rest either fire constantly and get muted, or never fire at all and get mistaken for coverage.</p>



<p class="wp-block-paragraph">This post is about building useful CloudWatch alarms: which signals deserve a page, which settings decide whether an alarm fires correctly, how to collapse alarm storms without going blind, and how to prove the notification path works before you need it.</p>



<h2 class="wp-block-heading">The test an alarm has to pass before it exists</h2>



<p class="wp-block-paragraph">Before the threshold, before the metric, one question: if this fires at 3am, what does the person on call do?</p>



<p class="wp-block-paragraph">If the honest answer is &#8220;look at it and go back to bed&#8221;, it is not an alarm. It is a dashboard panel. That single filter removes most of the CPU, memory and disk alarms people inherit. High CPU is not an incident. High CPU while latency climbs and the queue backs up is an incident. Every alarm that fires without requiring action trains the on-call to ignore the channel, and that training is permanent.</p>



<ul class="wp-block-list">
<li><strong>Page</strong> on symptoms a customer can feel: error rate, latency at a high percentile, queue age, failed jobs, expiring credentials.</li>

<li><strong>Ticket</strong> on causes that bite next week: disk trending toward full, approaching a service quota, a renewal that is not happening.</li>

<li><strong>Graph only</strong> on everything else. Resource utilisation is context during an incident, not the incident.</li>
</ul>



<p class="wp-block-paragraph">Encode that split in the alarm&#8217;s action, not in your head. A page goes to an SNS topic wired to PagerDuty or Opsgenie. A ticket goes to a different topic that opens a Jira issue or a Systems Manager OpsItem. Same engine, different consequences.</p>



<h2 class="wp-block-heading">The alarm that goes quiet: missing data is the invisible failure</h2>



<p class="wp-block-paragraph">Every metric alarm has a <code>TreatMissingData</code> setting, and if you never touched it, it is <code>missing</code>. That default means the alarm ignores gaps entirely when deciding state. It keeps whatever state it had, or sits in INSUFFICIENT_DATA if there is nothing at all. For a metric that always reports, that is a silent hole: the instance dies, the metric stops, and the alarm does not go red. It just stops having an opinion.</p>



<ul class="wp-block-list">
<li><code>breaching</code> for metrics that report continuously. If the data stops, something is wrong. Heartbeats, health checks, CPU on a box that should always be up.</li>

<li><code>notBreaching</code> for metrics that only exist when something bad happens. Error counts, throttled requests, dead letter queue depth. No data means no errors.</li>

<li><code>ignore</code> keeps the current state through gaps, so an alarm stays red until genuinely resolved rather than flapping to OK on a hole in the data.</li>

<li><code>missing</code> is the default, and defensible only when you decided it was right.</li>
</ul>



<p class="wp-block-paragraph">There is a second-order effect that surprises people. When an alarm evaluates, CloudWatch pulls a wider range of data points than the evaluation periods you configured. AWS calls that the evaluation range. If some points in that range are missing but enough real ones exist, CloudWatch evaluates against the real points and your missing-data setting never applies to the gaps.</p>



<p class="wp-block-paragraph">The consequence: an alarm set to eight breaching points out of ten can fire on one breaching point followed by nine gaps, because the older breach is still inside the evaluation range and everything newer is absent. Teams hit this with load balancer 5XX alarms constantly, and it looks exactly like an unexplained false positive.</p>



<p class="wp-block-paragraph">Setting <code>notBreaching</code> on sparse error metrics is usually the fix. Filling gaps with metric math is the other, and it is more explicit:</p>



<pre class="wp-block-code"><code># Treat absent error data points as zero rather than as gaps.
# m1 is the raw error metric; e1 is what the alarm evaluates.
e1 = FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">One exception: alarms on metrics in the <code>AWS/DynamoDB</code> namespace always ignore missing data regardless of what you set. If you need a DynamoDB alarm that reacts to silence, build the fill into metric math.</p>



<p class="wp-block-paragraph">Then find the ones already stuck. This is the fastest audit you can run on an inherited account:</p>



<pre class="wp-block-code"><code>aws cloudwatch describe-alarms 
  --state-value INSUFFICIENT_DATA 
  --query 'MetricAlarms[].[AlarmName,MetricName,StateReason]' 
  --output table</code></pre>



<p class="wp-block-paragraph">Anything on that list older than a deploy cycle is either watching something that no longer exists, or watching something that stopped reporting. Both deserve thirty seconds.</p>



<h2 class="wp-block-heading">The alarm that cries wolf: periods, data points and windows</h2>



<p class="wp-block-paragraph">Three settings decide whether an alarm is jumpy: <code>Period</code> (how long each data point covers), <code>EvaluationPeriods</code> (how many recent points to look at), and <code>DatapointsToAlarm</code> (how many of those must breach).</p>



<h3 class="wp-block-heading">M out of N beats consecutive breaches</h3>



<p class="wp-block-paragraph">Most people set evaluation periods to 3 and leave data points to alarm unset, which makes it 3 as well. That demands three consecutive breaching points. Real degradation is rarely that tidy; it flickers, one clean point resets your patience, and the alarm never fires.</p>



<p class="wp-block-paragraph">Set them separately. Three out of five catches intermittent problems while still filtering single-point spikes. The breaching points do not need to be consecutive, only inside the last N.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "api-5xx-elevated" 
  --namespace "AWS/ApplicationELB" 
  --metric-name HTTPCode_ELB_5XX_Count 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --statistic Sum 
  --period 60 
  --evaluation-periods 5 
  --datapoints-to-alarm 3 
  --threshold 25 
  --comparison-operator GreaterThanThreshold 
  --treat-missing-data notBreaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Read that as: over the last five minutes, if three or more individual minutes each saw more than 25 backend 5XX responses, page. Minutes with no data count as fine, because a load balancer with no errors publishes nothing.</p>



<h3 class="wp-block-heading">Percentiles need a sample-count decision</h3>



<p class="wp-block-paragraph">Average latency hides everything interesting, so alarm on <code>p95</code> or <code>p99</code> using the extended statistic rather than <code>--statistic Average</code>.</p>



<p class="wp-block-paragraph">The catch is low traffic. A p99 computed from four requests is noise, and by default CloudWatch still evaluates it. The <code>EvaluateLowSampleCountPercentile</code> setting controls that: set it to <code>ignore</code> and the alarm holds its current state when the sample count is too small to mean anything. On an overnight window or a quiet endpoint, that one setting removes a whole class of 4am pages.</p>



<h3 class="wp-block-heading">Sliding versus wall clock windows</h3>



<p class="wp-block-paragraph">By default the evaluation window slides: it advances a minute at a time and is not aligned to the clock. That is what you want for latency and errors, because you want to know now.</p>



<p class="wp-block-paragraph">CloudWatch also supports a wall clock window aligned to real boundaries in a time zone you pick: top of the hour, midnight, start of the calendar week. That is the right choice for anything measured per calendar period, like a daily batch job or a monthly threshold, because the alarm considers a completed period rather than a rolling one. Note that the wider evaluation range described earlier applies only to sliding windows, so wall clock alarms are easier to reason about but have no safety net for late-arriving data.</p>



<h2 class="wp-block-heading">Alarming on the wrong number</h2>



<p class="wp-block-paragraph">A large share of useless alarms are correctly configured alarms watching a meaningless quantity.</p>



<h3 class="wp-block-heading">Rates, not counts</h3>



<p class="wp-block-paragraph">An alarm on &#8220;more than 50 errors&#8221; is wrong twice. During a traffic peak, 50 errors out of half a million requests is nothing. At 3am, 50 errors out of 60 requests is a total outage that never crosses the threshold.</p>



<pre class="wp-block-code"><code># m1 = Errors (Sum), m2 = Invocations (Sum)
# Guard the zero case so the alarm gets a data point at idle
# instead of a gap it has to interpret.
e1 = IF(m2 &gt; 0, m1 / m2 * 100, 0)</code></pre>



<p class="wp-block-paragraph">Only the final expression should return data to the alarm; the underlying metrics are inputs. A rate alarm alone is still incomplete, though, because at very low volume one error is 100 percent. Pair it with a minimum-volume condition in a composite alarm, or fold the request count into the expression so tiny denominators cannot trip it.</p>



<h3 class="wp-block-heading">Quotas and saturation, before they bite</h3>



<p class="wp-block-paragraph">Service quotas are the outage nobody sees coming, because everything is healthy right up until it is not. For services that publish usage metrics, the <code>SERVICE_QUOTA</code> metric math function pulls your current limit from Service Quotas, so you can alarm on percentage consumed instead of a number you hardcoded and forgot:</p>



<pre class="wp-block-code"><code># m1 is a usage metric under the Usage namespace.
# Alarm on consumption as a share of the live quota.
e1 = m1 / SERVICE_QUOTA(m1) * 100</code></pre>



<p class="wp-block-paragraph">This is the definitive ticket-not-page alarm. Nobody needs waking for concurrency at 80 percent, but somebody needs to file a limit increase before the deploy that takes it to 100. Same category: queue age rather than queue depth. A deep queue draining fast is fine; a shallow queue whose oldest message is twenty minutes old means your consumers are stuck.</p>



<h2 class="wp-block-heading">Cutting the storm: composite alarms, suppression and mute rules</h2>



<p class="wp-block-paragraph">One bad deploy in an Auto Scaling group with per-instance alarms produces forty notifications describing one event. A composite alarm evaluates a boolean rule over other alarms. The metric alarms stay exactly as they are and keep their own state, but stop being wired to your paging topic. The composite becomes the only thing that pages.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Latency alone might be a slow dependency. Errors alone might be a bad client. Both together is a real problem, and that is what earns a page.</p>



<p class="wp-block-paragraph">The second half is suppression. A composite alarm accepts a suppressor alarm: while that alarm is in ALARM, the composite still evaluates but takes no actions. Point it at a deployment-in-progress alarm and you stop paging for known-bad windows.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --actions-suppressor "deployment-in-progress" 
  --actions-suppressor-wait-period 60 
  --actions-suppressor-extension-period 60 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page</code></pre>



<p class="wp-block-paragraph">Those two periods are the part people skip and then get burned by. The wait period gives the suppressor time to reach ALARM before the composite decides to fire; the extension period gives the composite time to settle back to OK after the suppressor clears. AWS suggests 60 seconds for each, on the reasoning that metric alarms evaluate once a minute. Leave them at zero and you get paged in the gap between the deploy starting and the deploy alarm noticing.</p>



<p class="wp-block-paragraph">CloudWatch has since added alarm mute rules, which handle the scheduled case directly. A mute rule targets specific alarms by name and mutes their actions during a one-time or recurring window. The alarms keep evaluating and keep changing state; only the actions are muted. That removes the EventBridge and Lambda contraption teams used to build for maintenance windows. Two things to keep straight: mute rules take precedence over composite action suppression when both apply, and a mute rule targets alarms by name, so a rename in your Terraform or OpenTofu module silently orphans the rule.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">Suppression is for known-bad windows you chose. If you are muting an alarm because it is annoying rather than because you scheduled the disruption, the alarm is wrong and muting it just hides that.</p>
</blockquote>



<h2 class="wp-block-heading">Test the path, not the threshold</h2>



<p class="wp-block-paragraph">An alarm you have never seen fire is a hypothesis. The threshold is the easy part. The fragile part is everything downstream: SNS subscriptions that were never confirmed, a Lambda whose role lost a permission, a chat webhook rotated three months ago, a topic policy that quietly rejects CloudWatch.</p>



<p class="wp-block-paragraph">You can force a state transition without touching the metric:</p>



<pre class="wp-block-code"><code>aws cloudwatch set-alarm-state 
  --alarm-name "api-5xx-elevated" 
  --state-value ALARM 
  --state-reason "Routing test, ticket OPS-1234"</code></pre>



<p class="wp-block-paragraph">This drives the real actions, so the page really goes out. Tell the on-call first. The forced state is temporary; the next real evaluation overwrites it. Do this for every new alarm before you call it done, and again after any change to the notification stack. On a mixed estate where some workloads sit on a VPS at a provider like Contabo or InterServer alongside your AWS footprint, test each path separately, because a shared SNS topic creates the illusion that one confirmed subscription covers everything.</p>



<p class="wp-block-paragraph">Then confirm the transition and the action both happened:</p>



<pre class="wp-block-code"><code>aws cloudwatch describe-alarm-history 
  --alarm-name "api-5xx-elevated" 
  --history-item-type Action 
  --max-records 10</code></pre>



<h2 class="wp-block-heading">What useful CloudWatch alarms cost you</h2>



<p class="wp-block-paragraph">Alarms bill per alarm metric per month, so the mental model that matters is this: a standard metric alarm counts as one, and a metric math alarm counts as one per metric referenced in the expression. An error-rate alarm built from errors and invocations bills as two, not one. Anomaly detection alarms carry their own multiplier because the band is evaluated alongside the metric. High-resolution alarms, with periods of 10, 20 or 30 seconds, are charged at a higher rate than standard ones, which makes them worth it for a handful of latency-critical paths and wasteful everywhere else.</p>



<p class="wp-block-paragraph">Check current rates on the CloudWatch pricing page rather than trusting a number in a blog post, including this one. And note the bigger cost is usually not the alarm at all. It is the custom metrics feeding it, especially high-cardinality ones published per request or per container.</p>



<h2 class="wp-block-heading">Troubleshooting an alarm that misbehaves</h2>



<ol class="wp-block-list">
<li><strong>Stuck in INSUFFICIENT_DATA.</strong> Either the metric is not being published, or the alarm&#8217;s dimensions do not match the metric&#8217;s dimensions exactly. Dimension mismatch is the common one and it is silent: an alarm on a dimension set that never existed looks identical to an alarm on a dead resource.</li>

<li><strong>Fired but nobody heard.</strong> Check <code>describe-alarm-history</code> filtered to Action items. Transition present but action missing points at the SNS topic policy or the subscription. Both present points downstream of SNS.</li>

<li><strong>Fires with no visible breach on the graph.</strong> Almost always missing data plus the wider evaluation range. The state reason names how many points were breaching and how many were unknown.</li>

<li><strong>Flapping between OK and ALARM.</strong> The period is too short for the metric&#8217;s natural variance, or data points to alarm equals evaluation periods. Widen the window before raising the threshold; raising the threshold loses sensitivity you may want later.</li>

<li><strong>Composite alarm never fires.</strong> Confirm every child alarm named in the rule exists and is spelled correctly, then confirm no suppressor or mute rule is active. The console shows a mute indicator on affected alarms.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving <code>TreatMissingData</code> at the default because you did not know it existed.</li>

<li>Picking 80 percent CPU because it sounds like a threshold, on a service that normally runs at 75.</li>

<li>Alarming per instance in an Auto Scaling group instead of on the aggregate a customer experiences.</li>

<li>Wiring OK actions to the same paging channel, so every recovery is also an interruption.</li>

<li>Alarm descriptions that restate the alarm name. The description is the only context the responder gets at 3am, so put the runbook link there.</li>

<li>Anomaly detection on a metric with no stable pattern. The band is only as good as the history it learned from.</li>
</ul>



<h2 class="wp-block-heading">Best practices for useful CloudWatch alarms</h2>



<ul class="wp-block-list">
<li>Set <code>TreatMissingData</code> explicitly on every alarm, and write down why in the description.</li>

<li>Separate <code>DatapointsToAlarm</code> from <code>EvaluationPeriods</code>. M out of N is almost always better than N consecutive.</li>

<li>Observe a metric across a full weekly cycle before fixing its threshold. Weekend traffic is a different distribution.</li>

<li>Define alarms in code, in whatever you already use: Terraform, OpenTofu, CDK, CloudFormation. Console-created alarms drift and disappear.</li>

<li>Use anomaly detection for the signal and composite alarms for the logic. An anomalous shape plus real user impact is a page; an anomalous shape alone is a graph.</li>

<li>Review what fired last quarter. Any alarm that fired and was closed without action is a candidate for deletion. Send it to a Grafana panel instead.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How many CloudWatch alarms should a service have?</h3>



<p class="wp-block-paragraph">Fewer than you think. A handful of symptom alarms that page, plus a set of cause alarms that open tickets, covers most services. The number that matters is not how many exist but how many fired last quarter and led to somebody doing something.</p>



<h3 class="wp-block-heading">Should I use anomaly detection or static thresholds?</h3>



<p class="wp-block-paragraph">Static thresholds where a real limit exists: a disk that must not fill, a latency budget you promised. Anomaly detection where the metric has a genuine daily or weekly shape and no fixed correct value, like request volume. Anomaly detection needs history before it is useful and struggles when a deploy changes the pattern, so it is a poor fit for anything new.</p>



<h3 class="wp-block-heading">Why did my alarm fire when the graph looks fine?</h3>



<p class="wp-block-paragraph">Usually missing data. CloudWatch pulls a wider evaluation range than the periods you configured, so an older breaching point can still count while newer points are simply absent. Read the state reason, which spells out how many points were breaching and how many were unknown, then reconsider the missing-data setting.</p>



<h3 class="wp-block-heading">Can an alarm watch a metric in another AWS account?</h3>



<p class="wp-block-paragraph">Cross-account observability lets a monitoring account view and alarm on metrics from linked source accounts, which is the usual pattern for an AWS Organizations estate. The alternative is running alarms in each account and aggregating at the SNS or paging layer. That is more alarms to manage but keeps the blast radius per account smaller.</p>



<h3 class="wp-block-heading">How do I stop alarms firing during a deployment?</h3>



<p class="wp-block-paragraph">Two options. An alarm mute rule on a schedule, if your deploys are predictable. Or a composite alarm with a deployment-in-progress alarm as its actions suppressor, if they are not. The suppressor approach suits continuous delivery better because it reacts to the actual deploy rather than a calendar guess.</p>



<h3 class="wp-block-heading">Do CloudWatch alarms work on logs?</h3>



<p class="wp-block-paragraph">Indirectly. A metric filter on a log group turns matching log lines into a numeric metric, and you alarm on that metric like any other. Metric filters only emit data points when they match, so <code>notBreaching</code> is almost always the right missing-data setting for them.</p>



<h3 class="wp-block-heading">Is CloudWatch enough, or do I need Datadog or Grafana?</h3>



<p class="wp-block-paragraph">For a single-account AWS workload, CloudWatch alarms plus SNS handle alerting fine, and staying inside AWS avoids another integration to maintain. Third-party platforms earn their keep when you need correlation across non-AWS systems, richer on-call routing, or long metric retention with fast queries. The honest test is whether you are already spending engineering time rebuilding those features on top of CloudWatch.</p>



<h2 class="wp-block-heading">The one thing to keep</h2>



<p class="wp-block-paragraph">Useful CloudWatch alarms are not the ones with the cleverest thresholds. They are the ones where somebody decided, on purpose, what silence means.</p>



<p class="wp-block-paragraph">Every alarm in your account is making a claim about the absence of data, whether you configured that claim or inherited it. Go through them, set <code>TreatMissingData</code> deliberately, split data points from evaluation periods, delete the ones nobody acts on, and force a state transition on the ones you keep so you know the page lands. Six alarms that fire correctly beat sixty that nobody reads.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your alerting?</h2>



<p class="wp-block-paragraph">I work with teams on AWS monitoring that people actually trust. Typically that looks like:</p>



<ul class="wp-block-list">
<li>Auditing existing CloudWatch alarms and separating the ones that page, the ones that should open tickets, and the ones to delete.</li>

<li>Fixing missing-data and evaluation settings so alarms stop firing on gaps and stop going silent on dead metrics.</li>

<li>Building composite alarms, suppression and mute rules so one incident produces one page instead of forty.</li>

<li>Writing metric math for error rates, queue age and quota headroom instead of raw counts.</li>

<li>Codifying the whole alarm set in Terraform or OpenTofu so it survives the next person.</li>

<li>Wiring CloudWatch into Grafana dashboards and testing every notification path end to end.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me the output of <code>describe-alarms</code> for one account and I will tell you which alarms are load-bearing and which are decoration.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/useful-cloudwatch-alarms/">CloudWatch Alarms That Are Actually Useful (and the Ones to Delete)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</title>
		<link>https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Amazon Managed Grafana]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Metric Math]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Notification Policies]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=260</guid>

					<description><![CDATA[<p>A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. An analyst pings you: &#8220;the dashboard is showing last week&#8217;s numbers.&#8221; You open Grafana, look at the pipeline board, and every panel is calm. No red. No spikes. Error counts flat at zero.</p>



<p class="wp-block-paragraph">Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.</p>



<p class="wp-block-paragraph">That gap is the most common problem I see with <strong>Grafana monitoring for AWS data pipelines</strong>, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.</p>



<h2 class="wp-block-heading">Failure family one: the pipeline stopped and nothing turned red</h2>



<p class="wp-block-paragraph">Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.</p>



<p class="wp-block-paragraph">Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.</p>



<p class="wp-block-paragraph">So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?</p>



<h3 class="wp-block-heading">No Data and MissingSeries are not the same thing</h3>



<ul class="wp-block-list">
<li><strong>No Data</strong> means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named <code>DatasourceNoData</code>. That alert does not inherit your rule&#8217;s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.</li>



<li><strong>MissingSeries</strong> means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with <code>grafana_state_reason</code> set to <code>MissingSeries</code>.</li>
</ul>



<p class="wp-block-paragraph">Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful &#8220;back to normal&#8221; notification. Nobody investigates a resolution.</p>



<h3 class="wp-block-heading">The fix: stop inferring health from absence</h3>



<p class="wp-block-paragraph">Do not build the &#8220;did it run&#8221; signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count</code></pre>



<p class="wp-block-paragraph">Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now &#8220;did it run&#8221; has a positive signal instead of an inferred one.</p>



<p class="wp-block-paragraph">The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are <code>missing</code>, <code>ignore</code>, <code>breaching</code> and <code>notBreaching</code>, and for a heartbeat you want <code>breaching</code>.</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Now every period has a value, zero means &#8220;no run completed&#8221;, and a plain threshold behaves the way you expected. <code>FILL(m1, REPEAT)</code> and <code>FILL(m1, LINEAR)</code> carry the last known value or interpolate instead, which suits gauges better than heartbeats.</p>



<p class="wp-block-paragraph">On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches <code>alertname=DatasourceNoData</code>, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.</p>



<h2 class="wp-block-heading">Failure family two: you are monitoring the compute, not the data</h2>



<p class="wp-block-paragraph">The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.</p>



<p class="wp-block-paragraph"><strong>Freshness</strong> is time since the last successful completion, per pipeline. It maps directly onto &#8220;can I trust the report on my screen&#8221;, which is the only question your stakeholders are asking. Derive it from the heartbeat.</p>



<p class="wp-block-paragraph"><strong>Volume</strong> is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:</p>



<ul class="wp-block-list">
<li><code>glue.driver.aggregate.recordsRead</code> and <code>glue.driver.aggregate.bytesRead</code> for input volume</li>



<li><code>glue.driver.aggregate.numFailedTasks</code> for partial failures that do not fail the job</li>



<li><code>glue.driver.s3.filesystem.write_bytes</code> and the <code>glue.ALL</code> equivalents for output across executors</li>
</ul>



<p class="wp-block-paragraph">Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are <code>ExecutionsStarted</code> minus <code>ExecutionsSucceeded</code> from the <code>AWS/States</code> namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the <code>AmazonMWAA</code> namespace, remembering that <code>metrics.metrics_allow_list</code> and <code>metrics.metrics_block_list</code> control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for &#8220;something is failing repeatedly and nobody noticed&#8221;.</p>



<h2 class="wp-block-heading">Failure family three: the dashboard costs more than the pipeline</h2>



<p class="wp-block-paragraph">Grafana&#8217;s CloudWatch data source uses <code>GetMetricData</code> to fetch samples and <code>ListMetrics</code> when you pick dimensions in the editor. <code>GetMetricData</code> does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.</p>



<p class="wp-block-paragraph">Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.</p>



<p class="wp-block-paragraph">The levers, roughly in order of return:</p>



<ol class="wp-block-list">
<li>Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.</li>



<li>Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.</li>



<li>Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.</li>



<li>Use template variables instead of one duplicated board per team.</li>



<li>If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy <code>GetMetricData</code> usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.</li>
</ol>



<p class="wp-block-paragraph">To find which client is driving the spend, enable CloudTrail data events on <code>GetMetricData</code>. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.</p>



<h2 class="wp-block-heading">Failure family four: access, accounts and regions</h2>



<p class="wp-block-paragraph">Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.</p>



<p class="wp-block-paragraph">Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.</p>



<h2 class="wp-block-heading">Where to run Grafana for this</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about viewer count and who carries the operational load.</p>



<ul class="wp-block-list">
<li><strong>Amazon Managed Grafana</strong> bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.</li>



<li><strong>Grafana Cloud</strong> from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.</li>



<li><strong>Self-hosted</strong> is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana&#8217;s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.</li>
</ul>



<h2 class="wp-block-heading">Laying out a board people actually read</h2>



<p class="wp-block-paragraph">Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.</p>



<ol class="wp-block-list">
<li><strong>Freshness table.</strong> One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline&#8217;s actual SLA. Most viewers read only this row.</li>



<li><strong>Volume against baseline.</strong> Records or bytes written per run, overlaid with the same period a week ago.</li>



<li><strong>Failure counts and durations</strong>, grouped by stage so a slow extract is distinguishable from a slow load.</li>



<li><strong>Resource detail at the bottom.</strong> Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.</li>
</ol>



<p class="wp-block-paragraph">Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:</p>



<pre class="wp-block-code"><code>SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn</code></pre>



<p class="wp-block-paragraph">Each <code>GetMetricData</code> operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.</p>



<h2 class="wp-block-heading">Troubleshooting: why the panel is empty</h2>



<ol class="wp-block-list">
<li><strong>Check the region on the query, not the data source.</strong> Grafana allows a per-query region override and it is easy to leave one on default.</li>



<li><strong>Widen the time range to seven days.</strong> If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.</li>



<li><strong>Check the statistic.</strong> Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.</li>



<li><strong>Check dimensions match exactly.</strong> With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.</li>



<li><strong>Check job metrics are enabled.</strong> For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.</li>



<li><strong>Check IAM.</strong> A role with <code>ListMetrics</code> but not <code>GetMetricData</code> populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.</li>



<li><strong>Check for throttling.</strong> Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.</li>
</ol>



<p class="wp-block-paragraph">For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises <code>DatasourceError</code> on evaluation timeouts or repeated query failures, governed by the <code>evaluation_timeout</code> and <code>max_attempts</code> settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.</li>



<li>Leaving no-data handling on the default and never routing <code>DatasourceNoData</code>. The alert fires into the void.</li>



<li>Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.</li>



<li>Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent <code>GetMetricData</code> bill for metrics nobody reads.</li>



<li>Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.</li>



<li>Write down a freshness SLA per pipeline, then set the threshold from it. &#8220;Alert if stale&#8221; only means something once someone says how stale is too stale.</li>



<li>Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.</li>



<li>Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.</li>



<li>Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a plugin to connect Grafana to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.</p>



<h3 class="wp-block-heading">Why does my Grafana alert resolve itself when the pipeline is still broken?</h3>



<p class="wp-block-paragraph">Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with <code>grafana_state_reason</code> set to <code>MissingSeries</code>. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.</p>



<h3 class="wp-block-heading">CloudWatch dashboards or Grafana for AWS data pipelines?</h3>



<p class="wp-block-paragraph">CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.</p>



<h3 class="wp-block-heading">How do I monitor a Glue job that succeeds but writes no data?</h3>



<p class="wp-block-paragraph">Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.</p>



<h3 class="wp-block-heading">Does Grafana querying CloudWatch cost money?</h3>



<p class="wp-block-paragraph">Yes. Grafana uses <code>GetMetricData</code>, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.</p>



<h3 class="wp-block-heading">Amazon Managed Grafana or self-hosted?</h3>



<p class="wp-block-paragraph">It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.</p>



<h3 class="wp-block-heading">What is the minimum useful monitoring for a new pipeline?</h3>



<p class="wp-block-paragraph">A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Effective <strong>Grafana monitoring for AWS data pipelines</strong> is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.</p>



<p class="wp-block-paragraph">Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.</p>



<h2 class="wp-block-heading">Need help building this on your AWS estate?</h2>



<p class="wp-block-paragraph">I design and build monitoring for AWS data platforms. This is the kind of work I take on:</p>



<ul class="wp-block-list">
<li>Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything</li>



<li>Grafana dashboards built around freshness and volume rather than executor charts nobody reads</li>



<li>Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity</li>



<li>CloudWatch query cost reviews, tracing <code>GetMetricData</code> spend back to the dashboard or rule causing it</li>



<li>Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys</li>



<li>Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-monitoring-aws-data-pipelines/">Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
