<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>FinOps | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/finops/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/finops/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 24 Oct 2026 06:02:47 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>FinOps | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/finops/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Parquet vs JSON vs CSV: What Each Format Actually Costs You</title>
		<link>https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 24 Oct 2026 06:02:47 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Compression]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CSV]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[JSON]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=339</guid>

					<description><![CDATA[<p>Format choice shows up as a bill, not a performance complaint. A working breakdown of where the money actually goes across Parquet, JSON and CSV: column pruning, row-group skipping, the gzip splittability trap, and request overhead. Includes a procedure for measuring the difference on your own data, and an honest account of where Parquet costs you.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;Athena spend tripled last month, nothing changed.&#8221; Nothing had changed on the query side. What changed was that an upstream team started dropping a wider export into the same S3 prefix, still as gzipped CSV, and every dashboard query that touched that table started reading every column of every file to answer a question about two of them.</p>



<p class="wp-block-paragraph">That is the failure mode that makes this comparison worth writing about. The <strong>Parquet vs JSON vs CSV</strong> question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.</p>



<p class="wp-block-paragraph">This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else&#8217;s benchmark, and an honest look at where CSV and JSON still beat the columnar option.</p>



<h2 class="wp-block-heading">The billing model is the whole story</h2>



<p class="wp-block-paragraph">Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.</p>



<p class="wp-block-paragraph">Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery&#8217;s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.</p>



<p class="wp-block-paragraph">That last part catches people. <code>LIMIT 10</code> does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which format is smaller on disk.&#8221; It is &#8220;how few bytes can the engine get away with reading.&#8221; Those diverge fast.</p>



<h2 class="wp-block-heading">Four levers, not one</h2>



<p class="wp-block-paragraph">When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.</p>



<h3 class="wp-block-heading">1. Column pruning</h3>



<p class="wp-block-paragraph">Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.</p>



<h3 class="wp-block-heading">2. Block skipping</h3>



<p class="wp-block-paragraph">Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.</p>



<p class="wp-block-paragraph">The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group&#8217;s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.</p>



<h3 class="wp-block-heading">3. Splittability</h3>



<p class="wp-block-paragraph">This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.</p>



<p class="wp-block-paragraph">Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.</p>



<p class="wp-block-paragraph">If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.</p>



<h3 class="wp-block-heading">4. Request overhead</h3>



<p class="wp-block-paragraph">Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine&#8217;s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">CSV: still the right answer more often than people admit</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.</p>



<p class="wp-block-paragraph">CSV is an excellent transport format and a poor storage format. That distinction is the useful one.</p>



<h2 class="wp-block-heading">JSON: the most expensive way to store a table</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Every record repeats every key name. Store ten million events with a field called <code>customer_identifier</code> and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.</p>



<p class="wp-block-paragraph">JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.</p>



<h2 class="wp-block-heading">Parquet: cheap reads, real obligations</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> This is the part the enthusiastic posts skip.</p>



<ul class="wp-block-list"><li><strong>You cannot append.</strong> Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.</li><li><strong>Schema mismatches across files are a runtime error, not a warning.</strong> Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.</li><li><strong>Timestamps are a compatibility minefield.</strong> The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.</li><li><strong>It is opaque.</strong> You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.</li><li><strong>Writing costs more.</strong> Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.</li></ul>



<p class="wp-block-paragraph">None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.</p>



<h2 class="wp-block-heading">Measure it on your data, in about twenty minutes</h2>



<p class="wp-block-paragraph">Published benchmarks tell you about somebody else&#8217;s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.</p>



<ol class="wp-block-list"><li><strong>Get a real byte count for what you have now.</strong> Not an estimate, the actual prefix.</li></ol>



<pre class="wp-block-code"><code>aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable</code></pre>



<p class="wp-block-paragraph">The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.</p>



<ol start="2" class="wp-block-list"><li><strong>Convert a representative slice, not the whole thing.</strong> One partition is enough. In Athena, a CTAS statement writes the output for you.</li></ol>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';</code></pre>



<p class="wp-block-paragraph">Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.</p>



<ol start="3" class="wp-block-list"><li><strong>Run your three most common queries against both tables and compare bytes scanned.</strong> Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.</li><li><strong>On BigQuery, price it before you run it.</strong> A dry run costs nothing and returns the byte estimate.</li></ol>



<pre class="wp-block-code"><code>bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'</code></pre>



<ol start="5" class="wp-block-list"><li><strong>Inspect the Parquet file you just wrote.</strong> Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.</li></ol>



<pre class="wp-block-code"><code>python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"</code></pre>



<p class="wp-block-paragraph">That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.</p>



<ol start="6" class="wp-block-list"><li><strong>Put a guardrail up before anyone else touches it.</strong> Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.</li></ol>



<p class="wp-block-paragraph">Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;We gzip our CSVs, so we already save the money.&#8221;</strong> Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.</li><li><strong>&#8220;Parquet is always faster.&#8221;</strong> Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.</li><li><strong>&#8220;We&#8217;ll just convert everything to Parquet.&#8221;</strong> Converting a landing zone that receives appended events all day means fighting the format&#8217;s biggest weakness. Land raw, convert on a schedule, query the converted copy.</li><li><strong>&#8220;Storage is cheap so format doesn&#8217;t matter.&#8221;</strong> Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.</li><li><strong>&#8220;Our BI tool caches, so nobody re-scans.&#8221;</strong> Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A short sequence of questions, in order.</p>



<ul class="wp-block-list"><li><strong>Is this data leaving your organisation?</strong> CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.</li><li><strong>Is it arriving continuously, one record at a time?</strong> Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.</li><li><strong>Is it read more than once, by more than one person, with filters?</strong> Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.</li><li><strong>Is the whole dataset smaller than a few hundred megabytes?</strong> Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.</li><li><strong>Are you fighting append semantics, concurrent writers, or schema evolution?</strong> You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.</li></ul>



<p class="wp-block-paragraph">The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV?</h3>



<p class="wp-block-paragraph">It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.</p>



<h3 class="wp-block-heading">Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?</h3>



<p class="wp-block-paragraph">Less than you would think, because once data is loaded into a BigQuery native table it lives in Google&#8217;s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.</p>



<h3 class="wp-block-heading">Is ORC better than Parquet?</h3>



<p class="wp-block-paragraph">They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet&#8217;s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.</p>



<h3 class="wp-block-heading">Why did my query cost the same after I converted to Parquet?</h3>



<p class="wp-block-paragraph">Usually one of three things. You are running <code>SELECT *</code>, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.</p>



<h3 class="wp-block-heading">Should I use Snappy, GZIP or ZSTD for Parquet?</h3>



<p class="wp-block-paragraph">Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.</p>



<h3 class="wp-block-heading">Can I query CSV and Parquet in the same table?</h3>



<p class="wp-block-paragraph">Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.</p>



<h3 class="wp-block-heading">What about the cost of the conversion job itself?</h3>



<p class="wp-block-paragraph">Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.</p>



<p class="wp-block-paragraph">Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help cutting a data lake bill?</h2>



<p class="wp-block-paragraph">Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves</li><li>Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses</li><li>Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control</li><li>Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges</li><li>Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types</li><li>Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end</li></ul>



<p class="wp-block-paragraph">If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Oracle on EC2 vs RDS: Why the License Math Doesn&#8217;t Save You</title>
		<link>https://john-nessime.com/blog/technical-guides/oracle-on-ec2-vs-rds/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 12 Oct 2026 06:08:11 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[BYOL]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Guard]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Multi-AZ]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[Oracle Licensing]]></category>
		<category><![CDATA[RDS Custom]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=386</guid>

					<description><![CDATA[<p>Oracle counts a vCPU the same way on EC2 and on RDS, so the platform choice never saves you licensing money. What it does change is sysdba access, block size and file limits, who owns backups and failover, and the Multi-AZ setting that quietly doubles your BYOL count.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/oracle-on-ec2-vs-rds/">Oracle on EC2 vs RDS: Why the License Math Doesn&#8217;t Save You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The finance lead asks a fair question in the migration kickoff: &#8220;If we move Oracle to RDS instead of EC2, how much do we save on licensing?&#8221;</p>



<p class="wp-block-paragraph">Nothing. Not a cent. Oracle counts a vCPU on Amazon RDS exactly the way it counts a vCPU on Amazon EC2, and it says so in the same paragraph of the same policy document. The managed service does not buy you a discount on the software.</p>



<p class="wp-block-paragraph">That single fact reshapes the whole argument. Once the license line is identical on both sides of the page, Oracle on EC2 vs RDS stops being a cost comparison and becomes a much more interesting question: which set of constraints can your application actually live inside? This post walks through the real differences, the places each option quietly hurts, the middle option that is being retired, and a decision procedure you can run before anyone opens a pricing calculator.</p>



<h2 class="wp-block-heading">Start with the thing that is not a difference</h2>



<p class="wp-block-paragraph">Oracle publishes a policy called &#8220;Licensing Oracle Software in the Cloud Computing Environment.&#8221; It names a short list of Authorized Cloud Environments, and both Amazon EC2 and Amazon RDS are on it, alongside Microsoft Azure and Google Cloud Platform. Inside an Authorized Cloud Environment the counting rules are these:</p>



<ul class="wp-block-list">
<li>Two vCPUs count as one Oracle Processor license when multi-threading of processor cores is enabled.</li>



<li>One vCPU counts as one Processor license when multi-threading is not enabled.</li>



<li>The Oracle Processor Core Factor Table does not apply. The 0.5 multiplier you rely on for x86 hardware in your own data center is simply gone.</li>



<li>For Standard Edition products, instances with four or fewer vCPUs count as one socket. Above four, every four vCPUs rounded up to the nearest multiple of four equals one socket.</li>



<li>Standard Edition 2 may only be licensed on instances up to eight AWS vCPUs. That is a hard ceiling, and it is the single most common reason an SE2 estate has to become an Enterprise Edition estate on migration.</li>
</ul>



<p class="wp-block-paragraph">None of those rules distinguish EC2 from RDS. The policy treats them as the same environment for counting purposes.</p>



<p class="wp-block-paragraph">There is one genuine licensing difference, and it is narrow. RDS offers a License Included model where AWS holds the Oracle license and bundles it into the instance-hour rate. That model covers Standard Edition 2 only. Enterprise Edition on RDS is bring-your-own-license, full stop. On EC2 there is no License Included option at all, so every Oracle database you run there is BYOL by definition.</p>



<p class="wp-block-paragraph">Worth reading the footer of that Oracle policy document, too. It states that the document is for educational purposes and may not be incorporated into any contract. Your Oracle Master Agreement and ordering documents are what actually bind you. The cloud policy is how Oracle says it will behave, not what it promised in writing, and it can change without notice. Pin a copy of the version in force when you deploy and file it with your architecture decision record.</p>



<h3 class="wp-block-heading">Find out what you are actually using first</h3>



<p class="wp-block-paragraph">Before you compare anything, find out which licensable options and management packs your current database is touching. Oracle tracks this itself. The feature usage view records every separately licensed feature the database has detected in use, which is exactly what an auditor looks at.</p>



<pre class="wp-block-code"><code>-- Which separately licensed features has this database touched?
SELECT name,
       detected_usages,
       currently_used,
       first_usage_date,
       last_usage_date
FROM   dba_feature_usage_statistics
WHERE  detected_usages &gt; 0
ORDER  BY name;</code></pre>



<p class="wp-block-paragraph">Oracle also publishes an options and packs usage script through My Oracle Support that goes further and correlates usage across several dictionary views. If you have a support contract, use that one rather than rolling your own. The output frequently surprises people. Partitioning, Advanced Compression, and the Diagnostics and Tuning Packs get switched on by a well-meaning DBA or by a default setting and then sit there generating an entitlement gap.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Amazon RDS for Oracle: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where RDS wins</h3>



<ul class="wp-block-list">
<li><strong>The operational floor is done for you.</strong> Automated backups with point-in-time recovery, minor version patching in a maintenance window, Multi-AZ synchronous standby with automatic failover, storage autoscaling. None of that needs a person on a rota.</li>



<li><strong>License Included removes the Oracle relationship entirely.</strong> For an SE2 workload where you own no licenses, this is genuinely hard to beat. There is no entitlement to track, no support renewal to chase, and no Oracle audit surface for the database software.</li>



<li><strong>Read replicas are configured for you.</strong> RDS manages the Data Guard setup behind cross-region replicas, including the network path between primary and standby. If the replica runs open read-only, that is Active Data Guard and you need the license for it, but the plumbing is not your problem.</li>



<li><strong>Parameter groups make configuration reviewable.</strong> Database parameters live in a versioned object you can diff, rather than in a spfile someone edited at two in the morning.</li>
</ul>



<h3 class="wp-block-heading">Where RDS hurts</h3>



<p class="wp-block-paragraph">The master user on an RDS for Oracle instance gets the DBA role with pieces removed. AWS documents exactly which privileges are withheld to keep the managed experience intact:</p>



<ul class="wp-block-list">
<li><code>ALTER DATABASE</code></li>



<li><code>ALTER SYSTEM</code></li>



<li><code>CREATE ANY DIRECTORY</code> and <code>DROP ANY DIRECTORY</code></li>



<li><code>GRANT ANY PRIVILEGE</code> and <code>GRANT ANY ROLE</code></li>
</ul>



<p class="wp-block-paragraph">You also cannot log in as <code>SYS</code>, <code>SYSTEM</code>, or any other Oracle-supplied administrative account. This is the wall most migrations hit, and they hit it late. A packaged vendor application ships an installer that expects to connect as sysdba, create a directory object pointing at a local path, and run <code>ALTER SYSTEM</code> a dozen times. On RDS that installer does not complete. There is no workaround, because the privilege is not withheld by policy, it is withheld by design.</p>



<p class="wp-block-paragraph">Three more constraints that bite later rather than sooner:</p>



<ul class="wp-block-list">
<li><strong>Block size is fixed at creation.</strong> RDS for Oracle instances are created with an 8 KB default block size and it cannot be changed afterwards. The SYSTEM and temporary tablespaces always use it. You can create additional tablespaces at other block sizes by allocating the matching cache parameter and using the <code>BLOCKSIZE</code> clause, but the core of the database is fixed.</li>



<li><strong>Single files cap at 16 TiB.</strong> That is the ext4 filesystem limit on the underlying host, so bigfile data files inherit it. Try to grow past it and you get an ORA-01237 with a &#8220;File too large&#8221; error underneath.</li>



<li><strong>No host access at all.</strong> No shell, no third-party backup agent, no monitoring agent installed on the database server, no custom scripts on the filesystem. If your compliance tooling or your APM vendor requires an on-host agent, that requirement decides the platform for you.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Oracle on EC2: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where EC2 wins</h3>



<ul class="wp-block-list">
<li><strong>Full sysdba and full root.</strong> Everything the vendor installer wants is available. Directory objects, external tables reading from local paths, custom PL/SQL that shells out, agents, whatever the packaged application assumed about its host.</li>



<li><strong>The full replication toolbox.</strong> Data Guard, Active Data Guard, and GoldenGate all run natively on EC2, and you control the protection mode, the apply lag tolerance, and the failover behaviour rather than accepting a managed default.</li>



<li><strong>Instance shape is a license lever.</strong> This is the underrated one. Oracle bills you on vCPUs, and most Oracle databases are memory bound rather than thread bound. On EC2 you can pick a memory-dense family and hit your SGA target without buying vCPUs, and therefore licenses, that the workload never uses. RDS gives you a narrower catalogue to choose from.</li>



<li><strong>You choose when to patch.</strong> No maintenance window imposed from outside, no forced minor version upgrade, no surprise reboot on a night your business cannot take one.</li>



<li><strong>Storage is yours to shape.</strong> Separate EBS volumes for data, redo, and archive logs, with independent IOPS and throughput. RDS abstracts that away.</li>
</ul>



<h3 class="wp-block-heading">Where EC2 hurts</h3>



<p class="wp-block-paragraph">Every one of those wins is a job someone now has to do, forever. The honest cost of Oracle on EC2 is not the instance-hour, it is the rota.</p>



<ul class="wp-block-list">
<li><strong>Backups you have to prove.</strong> RMAN to S3 is a solved pattern, but a backup that has never been restored is a hypothesis. Schedule a real restore into a throwaway instance on a cadence and treat a failed restore as an incident.</li>



<li><strong>A standby that drifts silently.</strong> Data Guard apply lag is the classic invisible failure. Everything looks green, the standby is mounted, redo is shipping, and apply stopped three days ago because of a single nologging operation or a filesystem that filled. You find out during the failover.</li>



<li><strong>Patching is now a project.</strong> Quarterly Oracle patches, OS patches, and the interaction between them. On a Data Guard pair that means a standby-first sequence and a rollback plan.</li>



<li><strong>Monitoring you have to build.</strong> Tablespace growth, apply lag, wait events, ASM disk group headroom, archive log destination fullness. Amazon CloudWatch gets you host metrics and nothing about the database. Teams typically land on a Prometheus exporter feeding Grafana, either self-hosted or through Grafana Cloud or Amazon Managed Grafana, plus alerting that someone actually answers.</li>



<li><strong>Cost attribution gets murky.</strong> Instance, EBS, snapshots, cross-AZ traffic, and the license amortisation all land in different places. A cost visibility tool such as Vantage or CloudZero earns its keep here, because the Oracle line is the one finance will ask about.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The middle option that is being retired</h2>



<p class="wp-block-paragraph">For a while the answer to &#8220;I need sysdba but I also want managed backups&#8221; was Amazon RDS Custom for Oracle. It gave you shell access to the underlying EC2 instance and sysdba on the database, while keeping automated backups and host recovery, policed by a mechanism called the support perimeter that flagged your instance as unsupported if a change broke the automation.</p>



<p class="wp-block-paragraph">AWS has announced that support for RDS Custom for Oracle ends on 31 March 2027. After that date the console and the resources go away, including instances, snapshots, and custom engine versions. AWS&#8217;s own guidance is to move those workloads to self-managed Oracle on EC2.</p>



<p class="wp-block-paragraph">This is one of the few dates worth putting in an article, because it removes an option from your decision tree. If you are choosing a platform today, RDS Custom for Oracle is not a candidate. If you are already on it, the migration is on your roadmap whether you have written it down or not. Note that this retirement does not touch RDS for Oracle or RDS Custom for SQL Server.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that actually move the number</h2>



<p class="wp-block-paragraph">Once you accept that the platform choice does not change the license rate, these are the things that do change the bill.</p>



<ol class="wp-block-list">
<li><strong>Memory-to-vCPU ratio.</strong> Reaching a memory target by moving up within a general purpose family buys vCPUs you did not need, and each pair of them is another Processor license. Picking a memory-optimised family instead is often the largest single saving available.</li>



<li><strong>Multi-AZ under BYOL doubles the count.</strong> AWS states this plainly in its own documentation: under BYOL you must have a license for both the primary and the standby DB instance in a Multi-AZ deployment. Enabling the checkbox on a BYOL instance without buying the second set of licenses puts you out of compliance the moment it comes up.</li>



<li><strong>Options and management packs.</strong> These are priced per Processor on top of the database license. On RDS, option groups can put some of them one click away, which is convenient and expensive in equal measure.</li>



<li><strong>Edition, forced by the vCPU ceiling.</strong> SE2 stops at eight vCPUs in an Authorized Cloud Environment. A workload that needs more compute has to move to Enterprise Edition, and that is a different order of magnitude, not a percentage increase.</li>



<li><strong>Infrastructure commitments.</strong> Reserved Instances and Savings Plans reduce the AWS side of the bill on both platforms. They do nothing to the Oracle side.</li>
</ol>



<p class="wp-block-paragraph">Two quick checks worth running before you model anything. First, confirm what an instance is actually presenting, because the multi-threading state is what decides whether you divide by two:</p>



<pre class="wp-block-code"><code># On an EC2 instance: threads per core tells you the multi-threading state.
# "Thread(s) per core: 2" means the 2 vCPU = 1 Processor rule applies.
lscpu | grep -E 'Thread(s) per core|Core(s) per socket|^CPU(s):'</code></pre>



<p class="wp-block-paragraph">Second, check what your existing RDS instances are declaring, since the license model and the Multi-AZ flag together determine your exposure:</p>



<pre class="wp-block-code"><code># Every Oracle RDS instance with its class, license model and Multi-AZ state.
aws rds describe-db-instances 
  --query "DBInstances[?starts_with(Engine, 'oracle')].
[DBInstanceIdentifier,Engine,DBInstanceClass,LicenseModel,MultiAZ]" 
  --output table</code></pre>



<p class="wp-block-paragraph">A <code>MultiAZ</code> value of <code>true</code> next to a <code>LicenseModel</code> of <code>bring-your-own-license</code> is the line to look at twice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How I would decide between Oracle on EC2 vs RDS</h2>



<p class="wp-block-paragraph">Run these in order. The first question that returns a hard &#8220;no&#8221; makes the decision for you, and you can stop.</p>



<ol class="wp-block-list">
<li><strong>Does anything need sysdba or host access?</strong> Vendor installers, on-host agents, directory objects pointing at local paths, external tables reading local files, custom OS-level jobs. If yes, it is EC2. Nothing else in this list matters.</li>



<li><strong>Run the feature usage query.</strong> Establish which options and packs you are on the hook for today. That number travels with you regardless of platform, and it is usually larger than the team expects.</li>



<li><strong>Check the physical constraints.</strong> Non-8 KB block size on the SYSTEM tablespace, or a single data file heading past 16 TiB, both rule RDS out.</li>



<li><strong>Settle the edition.</strong> If SE2 fits inside eight vCPUs and you own no perpetual licenses, RDS with License Included is very hard to argue against. If you need Enterprise Edition, you are BYOL on either platform and the licensing argument is over.</li>



<li><strong>Count licenses at the shape you would really run.</strong> Production instance, plus standby if you want Multi-AZ or Data Guard, plus every option pack from step two. Do this before you look at instance-hour rates, because it is the bigger number.</li>



<li><strong>Price the rota honestly.</strong> Who runs restore tests, who owns the patch calendar, who gets paged at three in the morning when apply lag alerts. If the answer is &#8220;we will figure it out,&#8221; that is a vote for RDS.</li>



<li><strong>Pick, then write down why.</strong> Record the Oracle policy version you counted against and the instance shapes you assumed. When an audit letter arrives, that document is the difference between a conversation and a negotiation.</li>
</ol>



<p class="wp-block-paragraph">One useful side effect of building a dev or reporting copy outside AWS entirely: a modest VPS from a provider like Contabo or InterServer running Oracle&#8217;s free Express Edition gives developers a schema to work against without touching a licensed instance. It will not stand in for production, but it takes pressure off the environments that cost money.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;RDS is cheaper because AWS manages it.&#8221;</strong> Under BYOL the license line is identical. What you save is operational labour, which is a real saving but a different budget line, and you should say which one you mean.</li>



<li><strong>&#8220;We will disable multi-threading to reduce licenses.&#8221;</strong> Turning it off halves your usable threads and flips the ratio to one vCPU per Processor license. You end up at the same license count with less compute.</li>



<li><strong>&#8220;The core factor gives us 0.5 on x86.&#8221;</strong> Not in an Authorized Cloud Environment. Oracle states the table does not apply there. This is the source of the widely repeated claim that moving to the cloud doubles Oracle licensing on identical hardware.</li>



<li><strong>&#8220;The standby is passive, so it is free.&#8221;</strong> AWS documents that BYOL Multi-AZ requires licensing both instances. There is a separate Oracle provision about genuinely cold failover nodes, but a synchronous standby that is always running is not that. Get your specific position confirmed in writing by whoever advises you on Oracle contracts, not from a blog, including this one.</li>



<li><strong>&#8220;Graviton is a different architecture, so it licenses differently.&#8221;</strong> The cloud policy counts vCPUs. It does not carve out processor architectures on shared tenancy.</li>



<li><strong>&#8220;We will start on RDS and move to EC2 if it does not work.&#8221;</strong> Technically true and practically expensive. Moving off RDS means a full migration project with a cutover window, not a configuration change. Establish the sysdba requirement before you build, not after.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Oracle licensing cheaper on RDS than on EC2?</h3>



<p class="wp-block-paragraph">Under bring-your-own-license, no. Oracle&#8217;s cloud policy names EC2 and RDS together and applies the same vCPU counting to both. The only structural difference is that RDS offers a License Included model, which is available for Standard Edition 2 only.</p>



<h3 class="wp-block-heading">Can I run Oracle Enterprise Edition with License Included on RDS?</h3>



<p class="wp-block-paragraph">No. AWS supports License Included for Standard Edition 2 only. Enterprise Edition on RDS is BYOL, and you need the appropriate licenses with active Software Update License and Support.</p>



<h3 class="wp-block-heading">Do I need to license the Multi-AZ standby?</h3>



<p class="wp-block-paragraph">Under BYOL, AWS documentation states you must have a license for both the primary and the standby DB instance. Budget for double the Processor count before you enable it. Under License Included the standby is reflected in the instance-hour rate instead.</p>



<h3 class="wp-block-heading">What is the largest Standard Edition 2 instance I can run on AWS?</h3>



<p class="wp-block-paragraph">Oracle&#8217;s cloud policy caps Standard Edition 2 at eight AWS vCPUs. Past that you are into Enterprise Edition. This ceiling catches a lot of migrations that were sized on memory requirements rather than vCPU count.</p>



<h3 class="wp-block-heading">Can I get sysdba access on RDS for Oracle?</h3>



<p class="wp-block-paragraph">No. The master user receives the DBA role minus a documented set of privileges including <code>ALTER SYSTEM</code>, <code>ALTER DATABASE</code>, <code>CREATE ANY DIRECTORY</code>, and the <code>GRANT ANY</code> privileges, and the Oracle-supplied administrative accounts are not available for login. If an application requires sysdba, EC2 is your platform.</p>



<h3 class="wp-block-heading">Is RDS Custom for Oracle still a viable middle ground?</h3>



<p class="wp-block-paragraph">Not for new work. AWS ends support for RDS Custom for Oracle on 31 March 2027 and recommends moving those workloads to EC2. Existing users should treat the migration as a scheduled project rather than a future problem.</p>



<h3 class="wp-block-heading">Does turning off hyperthreading reduce my Oracle license count?</h3>



<p class="wp-block-paragraph">No. With multi-threading enabled the ratio is two vCPUs per Processor license. With it disabled the ratio becomes one to one. The license requirement stays where it was while your available compute halves.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing from the Oracle on EC2 vs RDS comparison, make it this: the platform choice does not change what Oracle charges you. It changes who does the work and what your application is allowed to do.</p>



<p class="wp-block-paragraph">So run the decision in that order. Establish the sysdba requirement first, because it is binary and it ends the discussion. Count your options and packs second, because that number is usually wrong in the direction that costs money. Compare instance-hour rates last, because it is the smallest term in the equation and the one everyone starts with.</p>



<p class="wp-block-paragraph">And write down which version of Oracle&#8217;s cloud policy you counted against. It is a non-contractual document that can change, and the copy you filed on the day you deployed is worth more than any argument you can make from memory later.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on an Oracle migration to AWS?</h2>



<p class="wp-block-paragraph">Most of the expensive mistakes here happen before anything is built, in the assumptions nobody wrote down. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing feature and option pack usage on your existing databases so you know your real entitlement position before you size anything</li>



<li>Working through the sysdba and host-access requirements of packaged applications to determine whether RDS is even on the table</li>



<li>Modelling Processor counts across candidate instance shapes, including standby and Multi-AZ exposure, so the number in the business case survives review</li>



<li>Building the operational layer that self-managed Oracle on EC2 needs: RMAN to S3, tested restores, Data Guard apply-lag alerting, and patch runbooks</li>



<li>Planning migrations off RDS Custom for Oracle to EC2 ahead of the end-of-support date</li>



<li>Setting up monitoring and cost attribution that separates the AWS line from the Oracle line, so finance gets an answer instead of a total</li>
</ul>



<p class="wp-block-paragraph">Send me the output of your feature usage query, or a list of the instance classes you are considering, and I will tell you what I would check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/oracle-on-ec2-vs-rds/">Oracle on EC2 vs RDS: Why the License Math Doesn&#8217;t Save You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</title>
		<link>https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 23 Sep 2026 06:44:01 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost and Usage Report]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Log Retention]]></category>
		<category><![CDATA[Logs Insights]]></category>
		<category><![CDATA[Metric Filters]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Runaway Costs]]></category>
		<category><![CDATA[Showback]]></category>
		<category><![CDATA[VPC Flow Logs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=785</guid>

					<description><![CDATA[<p>CloudWatch rarely fails loudly, it accumulates. This is a breakdown of the four independent meters behind the bill: ingestion, storage, Logs Insights scans and custom metric cardinality. Why the same log line gets billed three times, why log groups never expire by default, how a single dimension can multiply your metrics bill without your traffic changing, and how to attribute the spend to specific log groups before you start deleting things.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in finance forwards the AWS invoice with a single line highlighted. AmazonCloudWatch, sitting uncomfortably close to your compute spend. Nobody launched anything. Nobody enabled a new feature. The number just kept climbing, in a slope gentle enough that nobody noticed until it was big enough to be somebody&#8217;s problem.</p>



<p class="wp-block-paragraph">That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.</p>



<p class="wp-block-paragraph">This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.</p>



<h2 class="wp-block-heading">The four meters behind your CloudWatch bill</h2>



<p class="wp-block-paragraph">CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.</p>



<ul class="wp-block-list">
<li><strong>Ingestion.</strong> Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.</li>

<li><strong>Storage.</strong> Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.</li>

<li><strong>Analysis.</strong> Logs Insights bills on data scanned by a query, not on rows returned and not per query.</li>

<li><strong>Metrics and API calls.</strong> Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.</li>
</ul>



<p class="wp-block-paragraph">The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.</p>



<h2 class="wp-block-heading">Ingestion is the meter that actually hurts</h2>



<h3 class="wp-block-heading">Uncompressed in, compressed out</h3>



<p class="wp-block-paragraph">Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.</p>



<p class="wp-block-paragraph">That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The <code>storedBytes</code> value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.</p>



<h3 class="wp-block-heading">The log groups you never created</h3>



<p class="wp-block-paragraph">Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.</p>



<p class="wp-block-paragraph">None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.</p>



<p class="wp-block-paragraph">The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.</p>



<h2 class="wp-block-heading">Retention: the one-line fix almost nobody applies</h2>



<p class="wp-block-paragraph">Log groups default to <strong>Never Expire</strong>. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.</p>



<p class="wp-block-paragraph">Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The <code>?!retentionInDays</code> filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit &#8220;never&#8221; value to match on.</p>



<pre class="wp-block-code"><code>aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Three things are worth knowing before running that across an account.</p>



<ol class="wp-block-list">
<li><strong>The value is an enum, not a free integer.</strong> Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.</li>

<li><strong>Deletion is not instant.</strong> Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported <code>storedBytes</code>, so the billing effect lands before the data physically goes.</li>

<li><strong>Reverting means removing the policy, not setting zero.</strong> To go back to indefinite retention you call <code>delete-retention-policy</code> rather than passing a zero value.</li>
</ol>



<p class="wp-block-paragraph">Dry run before you bulk-apply. This prints what would change without touching anything:</p>



<pre class="wp-block-code"><code>aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done</code></pre>



<p class="wp-block-paragraph">Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.</p>



<h2 class="wp-block-heading">Cardinality: the multiplier nobody sees coming</h2>



<p class="wp-block-paragraph">This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.</p>



<p class="wp-block-paragraph">CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish <code>OrderLatency</code> with a <code>Region</code> dimension holding three values and you have three metrics. Add a <code>CustomerId</code> dimension and you have three times however many customers you have.</p>



<p class="wp-block-paragraph">The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.</p>



<h3 class="wp-block-heading">Metric filters have a guard rail, and it bites</h3>



<p class="wp-block-paragraph">Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.</p>



<p class="wp-block-paragraph">That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.</p>



<p class="wp-block-paragraph">Audit what exists before adding more, and count what a namespace actually produces:</p>



<pre class="wp-block-code"><code>aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'</code></pre>



<p class="wp-block-paragraph">If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.</p>



<h3 class="wp-block-heading">EMF does exactly what you tell it</h3>



<p class="wp-block-paragraph">Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.</p>



<p class="wp-block-paragraph">The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.</p>



<h3 class="wp-block-heading">Anomaly detection alarms cost more than they look</h3>



<p class="wp-block-paragraph">An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.</p>



<p class="wp-block-paragraph">It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it &#8220;to be safe&#8221; triples a line item for very little detection benefit.</p>



<h2 class="wp-block-heading">Query cost spikes exactly when you need it most</h2>



<p class="wp-block-paragraph">Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.</p>



<p class="wp-block-paragraph">During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:</p>



<ul class="wp-block-list">
<li><strong>Narrow the time window first, then the filter.</strong> Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.</li>

<li><strong>Select specific log groups, not patterns.</strong> Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.</li>
</ul>



<p class="wp-block-paragraph">Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.</p>



<h2 class="wp-block-heading">Finding out where your CloudWatch costs come from</h2>



<p class="wp-block-paragraph">Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.</p>



<ol class="wp-block-list">
<li><strong>Split the bill by meter.</strong> In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: <code>DataProcessing-Bytes</code> is log ingestion, <code>TimedStorage-ByteHrs</code> is log storage, <code>DataScanned-Bytes</code> is Logs Insights, and <code>CW:Requests</code> and <code>CW:GMD-Metrics</code> cover API calls.</li>

<li><strong>Rank log groups by ingestion.</strong> The <code>IncomingBytes</code> metric in the <code>AWS/Logs</code> namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an <em>absolute</em> range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.</li>

<li><strong>Check stored bytes separately.</strong> A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.</li>

<li><strong>Count custom metrics per namespace.</strong> Compare the count against your mental model of what you measure. The gap is cardinality.</li>

<li><strong>Look inside the noisiest group.</strong> Once you know which log group dominates, find out which streams are responsible before changing any application code.</li>
</ol>



<p class="wp-block-paragraph">For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:</p>



<pre class="wp-block-code"><code>stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20</code></pre>



<p class="wp-block-paragraph">That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.</p>



<p class="wp-block-paragraph">If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.</p>



<h2 class="wp-block-heading">Cheaper destinations, and what they cost you</h2>



<p class="wp-block-paragraph">Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.</p>



<ul class="wp-block-list">
<li><strong>The Infrequent Access log class.</strong> Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.</li>

<li><strong>Subscription filter to S3 via Amazon Data Firehose.</strong> Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.</li>

<li><strong>A separate log stack.</strong> Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.</li>
</ul>



<p class="wp-block-paragraph">The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.</p>



<h2 class="wp-block-heading">Troubleshooting a bill that will not go down</h2>



<p class="wp-block-paragraph"><strong>You set retention everywhere and storage barely moved.</strong> Check whether storage was ever the dominant cost. If your usage type breakdown was mostly <code>DataProcessing-Bytes</code>, retention was never going to fix it.</p>



<p class="wp-block-paragraph"><strong>Custom metric count grows with no code changes.</strong> Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.</p>



<p class="wp-block-paragraph"><strong>An alarm went quiet and the bill jumped the same week.</strong> Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm&#8217;s missing data treatment. Alarms treating missing data as not breaching fail silently by design.</p>



<p class="wp-block-paragraph"><strong>API request charges you cannot place.</strong> External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.</p>



<p class="wp-block-paragraph"><strong>Costs dropped, then came back.</strong> Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.</li>

<li>Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.</li>

<li>Filtering logs after they reach CloudWatch. The ingestion charge already applied.</li>

<li>Deleting log groups to cut cost without checking whether anything alarms on them.</li>

<li>Setting retention in the console while the IaC that owns the log group has none configured.</li>

<li>Enabling every EKS control plane log type when only the audit log is consumed.</li>
</ul>



<h2 class="wp-block-heading">Best practices for controlling CloudWatch costs</h2>



<ul class="wp-block-list">
<li><strong>Make retention a required argument.</strong> If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.</li>

<li><strong>Decide verbosity per environment, not per service.</strong> Debug in staging, info in production, with a documented way to raise it temporarily during an incident.</li>

<li><strong>Keep dimensions bounded, identifiers in the message body.</strong> If you cannot enumerate a dimension&#8217;s possible values on a whiteboard, it does not belong in the dimension set.</li>

<li><strong>Alarm on the bill itself.</strong> A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.</li>

<li><strong>Review the top five log groups by ingestion monthly.</strong> Five minutes with the <code>IncomingBytes</code> graph catches new noise before it compounds.</li>

<li><strong>Route archival logs out deliberately.</strong> Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why is my CloudWatch bill so high when my traffic has not changed?</h3>



<p class="wp-block-paragraph">Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.</p>



<h3 class="wp-block-heading">Does deleting a log group reduce my bill immediately?</h3>



<p class="wp-block-paragraph">It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.</p>



<h3 class="wp-block-heading">What counts as one custom metric for billing?</h3>



<p class="wp-block-paragraph">One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.</p>



<h3 class="wp-block-heading">Should I switch log groups to the Infrequent Access class?</h3>



<p class="wp-block-paragraph">For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.</p>



<h3 class="wp-block-heading">Is Logs Insights charged per query or per gigabyte?</h3>



<p class="wp-block-paragraph">Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.</p>



<h3 class="wp-block-heading">Do VPC Flow Logs have to go to CloudWatch?</h3>



<p class="wp-block-paragraph">No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.</p>



<p class="wp-block-paragraph">So attribute first. Split by usage type, rank log groups by <code>IncomingBytes</code>, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.</p>



<h2 class="wp-block-heading">Need a hand with your CloudWatch spend?</h2>



<p class="wp-block-paragraph">This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:</p>



<ul class="wp-block-list">
<li>Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes</li>

<li>Auditing custom metric cardinality and identifying which dimensions are multiplying your series count</li>

<li>Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data</li>

<li>Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment</li>

<li>Designing a subscription filter and S3 archive path for logs you need to keep but rarely query</li>

<li>Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of <code>describe-log-groups</code> for the account. That is usually enough to say where the money is going before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudwatch-costs-logs-metrics-cardinality/">The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</title>
		<link>https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 15 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost and Usage Report]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Service Control Policies]]></category>
		<category><![CDATA[Showback]]></category>
		<category><![CDATA[Tag Policies]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=519</guid>

					<description><![CDATA[<p>Tagging a resource and activating a cost allocation tag are two different operations, and AWS never tells you when you have only done the first. A practical guide to the activation gap, what backfill can and cannot repair, the five places coverage leaks, and a retrofit order that actually converges.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/">AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You tagged everything. The Terraform provider block carries <code>default_tags</code> with <code>Project</code> and <code>Environment</code>, every module inherits them, and you can open the console right now and see those tags sitting on the instances.</p>



<p class="wp-block-paragraph">Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by <code>Project</code>, and over half the bill lands in a bucket called &#8220;No tag key&#8221;.</p>



<p class="wp-block-paragraph">Nothing is broken. The tags are there. They just aren&#8217;t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you&#8217;ve done only the first.</p>



<p class="wp-block-paragraph">This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won&#8217;t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.</p>



<h2 class="wp-block-heading">A resource tag and a cost allocation tag are not the same thing</h2>



<p class="wp-block-paragraph">A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.</p>



<p class="wp-block-paragraph">Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that&#8217;s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.</p>



<p class="wp-block-paragraph">Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between &#8220;we agreed a standard&#8221; and &#8220;I can see it in Cost Explorer&#8221; is two days at best.</p>



<p class="wp-block-paragraph">Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:</p>



<pre class="wp-block-code"><code># Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'</code></pre>



<p class="wp-block-paragraph">User-defined keys appear in the report with a <code>user:</code> prefix and AWS-generated ones carry the reserved <code>aws:</code> prefix, so they never collide. There&#8217;s a cap on active keys, raisable through Service Quotas, which you almost certainly don&#8217;t need to raise.</p>



<h2 class="wp-block-heading">What the backfill repairs, and what it quietly doesn&#8217;t</h2>



<p class="wp-block-paragraph">Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That&#8217;s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.</p>



<pre class="wp-block-code"><code># Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z</code></pre>



<ul class="wp-block-list">
<li>Start date must be the first day of a month, and can&#8217;t reach back more than twelve months.</li>

<li>One request per 24 hours, and you can&#8217;t submit another while one is processing.</li>

<li>It backfills <em>activation status</em>, not tag values.</li>
</ul>



<p class="wp-block-paragraph">That last point catches people. If the resource wasn&#8217;t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes &#8220;I forgot to activate.&#8221; It does not fix &#8220;I forgot to tag.&#8221; A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.</p>



<p class="wp-block-paragraph">It also cuts both ways. Backfill applies the <em>current</em> status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you&#8217;re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you&#8217;ll erase the history you were trying to keep.</p>



<h2 class="wp-block-heading">Where AWS cost allocation tags actually leak coverage</h2>



<p class="wp-block-paragraph">With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.</p>



<h3 class="wp-block-heading">Charges no resource can carry</h3>



<p class="wp-block-paragraph">Support charges, refunds, credits, tax, some Marketplace subscriptions. There&#8217;s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that&#8217;s a visible fraction of the invoice.</p>



<p class="wp-block-paragraph">The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can&#8217;t reach.</p>



<p class="wp-block-paragraph">This is recent enough that most tagging guides predate it, and it&#8217;s the highest-value item here if your accounts already map roughly onto teams.</p>



<h3 class="wp-block-heading">Resources created by other resources</h3>



<p class="wp-block-paragraph">This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform&#8217;s provider-level <code>default_tags</code> applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn&#8217;t create those, the ASG did. Your ASG is tagged. Your instances aren&#8217;t. Same for volumes attached at launch, ENIs, snapshots and AMIs.</p>



<pre class="wp-block-code"><code>data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}</code></pre>



<p class="wp-block-paragraph">For a standalone instance, <code>volume_tags</code> on <code>aws_instance</code> covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don&#8217;t propagate until you&#8217;ve checked the created resource in the console, not the plan output.</p>



<h3 class="wp-block-heading">Shared infrastructure that belongs to everyone</h3>



<p class="wp-block-paragraph">NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says <code>Project=platform</code>, which is accurate and useless for chargeback.</p>



<p class="wp-block-paragraph">Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.</p>



<p class="wp-block-paragraph">Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.</p>



<h3 class="wp-block-heading">Container spend hiding behind one instance tag</h3>



<p class="wp-block-paragraph">A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams&#8217; workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won&#8217;t account for idle headroom you&#8217;re deliberately carrying, but it beats one line item labelled &#8220;the cluster&#8221;.</p>



<h3 class="wp-block-heading">Case drift and near-duplicate keys</h3>



<p class="wp-block-paragraph">Keys and values are case-sensitive. <code>Environment</code>, <code>environment</code> and <code>ENV</code> are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A retrofit order that converges</h2>



<p class="wp-block-paragraph">Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.</p>



<ol class="wp-block-list">
<li><strong>Measure first.</strong> Group Cost Explorer by your intended key and record what percentage of spend falls into &#8220;No tag key&#8221;. That&#8217;s your baseline and your only honest progress metric.</li>

<li><strong>Cut the key set down.</strong> Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.</li>

<li><strong>Activate immediately, even with terrible coverage.</strong> Both clocks start now: the activation delay and the rolling twelve-month backfill window. There&#8217;s no downside to activating a key that&#8217;s barely applied.</li>

<li><strong>Rank the gap by spend, not resource count.</strong> Sort untagged line items by cost descending. It&#8217;s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.</li>

<li><strong>Fix the creation path before the backlog.</strong> IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.</li>

<li><strong>Sweep the stragglers in bulk.</strong> The Resource Groups Tagging API works across services and regions, which beats going service by service.</li>

<li><strong>Backfill, then re-measure.</strong> Compare against step one. If the number barely moved, you fixed the wrong resources.</li>
</ol>



<pre class="wp-block-code"><code># Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod</code></pre>



<p class="wp-block-paragraph">Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.</p>



<h2 class="wp-block-heading">Enforcing tags without becoming the tag police</h2>



<p class="wp-block-paragraph">Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.</p>



<ul class="wp-block-list">
<li><strong>IaC defaults.</strong> Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.</li>

<li><strong>CI validation.</strong> A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.</li>

<li><strong>Detective controls.</strong> The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.</li>

<li><strong>Preventive controls.</strong> A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.</li>
</ul>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody&#8217;s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone&#8217;s appetite for the project.</p>



<p class="wp-block-paragraph">Scope it to one service at a time, starting in sandbox. I&#8217;d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.</p>



<h2 class="wp-block-heading">Measuring coverage so it doesn&#8217;t rot</h2>



<p class="wp-block-paragraph">Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a <code>DESCRIBE</code> first rather than guessing.</p>



<pre class="wp-block-code"><code>-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost &gt; 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;</code></pre>



<p class="wp-block-paragraph">That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.</p>



<p class="wp-block-paragraph">If you&#8217;d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>The key isn&#8217;t in the activation list.</strong> It must exist on at least one resource first, and propagation takes up to a day. Past that, check you&#8217;re in the management account and the key matches exactly, including case.</li>

<li><strong>Activated, but everything still shows untagged.</strong> Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.</li>

<li><strong>Tags stopped working after an account move.</strong> Activation doesn&#8217;t survive a move between organizations. Reactivate from the new management account, then backfill.</li>

<li><strong>Cluster costs still land on one team.</strong> Instance tags can&#8217;t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.</li>

<li><strong>You&#8217;ve hit the active key limit.</strong> Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming a tagged resource is a tagged cost. It isn&#8217;t until you activate the key.</li>

<li>Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.</li>

<li>Deactivating stale keys and then backfilling, which wipes their history out of the data.</li>

<li>Measuring coverage by resource count instead of by spend.</li>

<li>Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.</li>

<li>Trusting <code>default_tags</code> to reach resources Terraform didn&#8217;t directly create.</li>

<li>Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep the mandatory set to three to five keys, with defined allowed values rather than free text.</li>

<li>Pin casing in the standard and enforce it with Tag Policies, because AWS won&#8217;t do it for you.</li>

<li>Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.</li>

<li>Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.</li>

<li>Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.</li>

<li>Put allocation rules in Cost Categories rather than inventing tag values that encode a split.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Are AWS cost allocation tags retroactive?</h3>



<p class="wp-block-paragraph">Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.</p>



<h3 class="wp-block-heading">Why don&#8217;t my tags appear in Cost Explorer?</h3>



<p class="wp-block-paragraph">In order of likelihood: the key was never activated in the Billing console; you&#8217;re looking from a member account; the activation delay hasn&#8217;t elapsed; or the key differs in case from the one you activated.</p>



<h3 class="wp-block-heading">How many cost allocation tags can I activate?</h3>



<p class="wp-block-paragraph">There&#8217;s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.</p>



<h3 class="wp-block-heading">Should I use account tags or resource tags?</h3>



<p class="wp-block-paragraph">Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.</p>



<h3 class="wp-block-heading">Is this worth doing on a small account?</h3>



<p class="wp-block-paragraph">Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.</p>



<h3 class="wp-block-heading">How do I allocate shared costs like NAT gateways?</h3>



<p class="wp-block-paragraph">Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS cost allocation tags fail silently, and that&#8217;s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can&#8217;t answer.</p>



<p class="wp-block-paragraph">So check one thing now: open the cost allocation tags page in your management account and see whether the keys you&#8217;ve been diligently applying are actually switched on. If they aren&#8217;t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can&#8217;t recover.</p>



<p class="wp-block-paragraph">Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your AWS bill to explain itself?</h2>



<p class="wp-block-paragraph">Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:</p>



<ul class="wp-block-list">
<li>Auditing tag coverage as a percentage of spend, broken down by service, so you know what you&#8217;re actually fixing</li>

<li>Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization</li>

<li>Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage</li>

<li>Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team&#8217;s number</li>

<li>Building cost and usage export pipelines in Athena, partitioned so the dashboard isn&#8217;t expensive to load</li>

<li>Planning a staged tag enforcement rollout that doesn&#8217;t break anyone&#8217;s deployment pipeline</li>
</ul>



<p class="wp-block-paragraph">Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I&#8217;ll tell you where the gap actually is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-cost-allocation-tags-coverage/">AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From</title>
		<link>https://john-nessime.com/blog/cloud-computing/cloudflare-r2-vs-amazon-s3-media-offloading/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 10 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Web Performance]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cache Control]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[CDN]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Residency]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Object Cache]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[Website Performance]]></category>
		<category><![CDATA[WP-CLI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=382</guid>

					<description><![CDATA[<p>Free egress is not free serving. R2 still bills every read, S3 does not bill origin transfer to CloudFront, and the variable that decides both bills is cache hit ratio. An evenhanded comparison of Cloudflare R2 vs Amazon S3 for media offloading, with the four levers that move the number and a decision procedure you can run against your own traffic.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudflare-r2-vs-amazon-s3-media-offloading/">Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The disk-full alert was the easy part. A media library outgrows the box it lives on, someone moves it to object storage over a weekend, the alert stops firing, everyone moves on. Two weeks later the complaint is different: images are slow for visitors on the other side of the planet, and there is a line on the bill nobody can explain item by item.</p>



<p class="wp-block-paragraph">Moving the files is the solved part. Which bucket they land in, and what sits in front of that bucket, decides whether you spend almost nothing or spend more than the server you were trying to shrink.</p>



<p class="wp-block-paragraph">Cloudflare R2 vs Amazon S3 usually gets reduced to one sentence: R2 does not charge for data transfer out, S3 does, therefore R2. That is wrong in both directions, and understanding why is most of the value here. This post covers the failure mode that shows up after the migration rather than during it, an honest profile of each service, the levers that actually move the number, and a procedure you can run against your own traffic in an afternoon.</p>



<h2 class="wp-block-heading">Free egress is not free serving</h2>



<p class="wp-block-paragraph">R2 charges nothing to move bytes out, on any storage class. It still charges for the read. Every <code>GetObject</code> and every <code>HeadObject</code> is a Class B operation, and a media library made of thousands of small files racks those up fast. Cloudflare&#8217;s own asset hosting example in the R2 pricing docs makes the point: storage falls inside the free tier, writes fall inside the free tier, and the whole bill is read operations.</p>



<p class="wp-block-paragraph">So the meter is not gigabytes. The meter is requests that reach the bucket, which means the variable deciding your bill on either platform is cache hit ratio.</p>



<p class="wp-block-paragraph">This is where offloading setups go wrong without anyone noticing. The plugin rewrites image URLs to point at the bucket, images render, pages look fine, and every request for every thumbnail travels all the way to the origin and gets billed. Nothing breaks. Nothing alerts. The number just grows with traffic.</p>



<p class="wp-block-paragraph">There is a sharper version on the R2 side. A new bucket gives you a public URL on an <code>r2.dev</code> subdomain, which is convenient and tempting. Cloudflare documents it as a testing endpoint with a variable rate limit. Push production traffic through it and requests get throttled with <code>429 Too Many Requests</code>, throughput can be throttled too, and you get no cache, no WAF, no bot management. Those only exist once the bucket sits behind a custom domain you control. That is the invisible failure: not a broken image, a working setup billed on every request and rate limited under load.</p>



<h2 class="wp-block-heading">Cloudflare R2: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where R2 wins</h3>



<ul class="wp-block-list">
<li><strong>Egress really is zero.</strong> Not zero within a partner network, not zero up to a ratio of stored data. Zero, on both storage classes, through the S3 API and through Workers.</li>

<li><strong>A custom domain puts it behind the CDN.</strong> One managed CNAME and reads start being absorbed at the edge instead of hitting the bucket. No second product to buy and wire up.</li>

<li><strong>Deletes are free operations</strong>, along with aborting a multipart upload. Cleaning up a messy library costs nothing.</li>

<li><strong>Unauthorized requests are not billed.</strong> A caller without permission gets a 401 and you are not charged, which matters once someone starts hammering your bucket path.</li>

<li><strong>Migration tooling is free to use.</strong> Super Slurper copies a bucket across in bulk, Sippy migrates lazily on first miss. You pay only for the operations they perform against R2.</li>
</ul>



<h3 class="wp-block-heading">Where R2 hurts</h3>



<ul class="wp-block-list">
<li><strong>S3 compatible is not S3 identical.</strong> Cloudflare publishes a table of which operations are implemented and which are not. Real tooling has broken on the gaps: when AWS shipped SDKs that enabled CRC32 checksums by default, R2 rejected the header until the mismatch was resolved.</li>

<li><strong>There is no cold archive tier.</strong> Infrequent Access is as cold as it gets, and it carries a minimum storage duration, a retrieval fee, and doubled operation rates. For an active library that is the wrong tier, not a saving.</li>

<li><strong>Data lives in one primary location.</strong> Location hints are best effort rather than a region guarantee, and are honored only the first time a bucket with that name is created. Read performance away from that location comes from the edge cache, not from replicas.</li>

<li><strong>Plugin support is thinner.</strong> WP Offload Media, the long-standing WordPress option, officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage. R2 users generally land on Media Cloud, Advanced Media Offloader, Next3 Offload, or the S3-Uploads route driven from WP-CLI.</li>
</ul>



<h2 class="wp-block-heading">Amazon S3: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where S3 wins</h3>



<ul class="wp-block-list">
<li><strong>Origin transfer to CloudFront is not billed.</strong> This quietly demolishes the usual comparison. AWS waives data transfer from an AWS origin to CloudFront, so a properly built S3 media stack never pays the direct-to-internet rate on cached traffic.</li>

<li><strong>The CloudFront free allowance is permanent</strong>, covering a monthly volume of data transfer out and requests rather than expiring after twelve months. Small sites can sit inside it indefinitely.</li>

<li><strong>Event driven derivatives are mature.</strong> Upload fires a Lambda that generates thumbnails, strips metadata, converts formats, writes the result back. Hard to beat if your media pipeline does real work.</li>

<li><strong>A full storage ladder with real lifecycle transitions</strong>, down to deep archive. If a large chunk of your library is genuinely cold, S3 has somewhere cheap to park it.</li>

<li><strong>Governance depth.</strong> Object Lock, versioning, replication, IAM condition keys, CloudTrail data events. When an auditor asks for a control, S3 usually has a named feature for it.</li>
</ul>



<h3 class="wp-block-heading">Where S3 hurts</h3>



<ul class="wp-block-list">
<li><strong>The default configuration is the expensive one.</strong> Point image URLs at the bucket endpoint and every byte leaves at the internet rate with nothing cached in front of it. The cheap path has to be built deliberately.</li>

<li><strong>Four meters instead of two:</strong> storage, bucket requests, CDN requests, CDN data transfer, plus per-feature charges. Forecasting takes real effort.</li>

<li><strong>Geography changes the rate.</strong> Delivery costs more per gigabyte to some regions than others, so an audience shift moves your bill without you changing anything.</li>

<li><strong>Leaving costs money.</strong> Copying a large library out is billed transfer, once, in a lump. Usually small against ongoing savings, but it is a real number somebody has to approve.</li>
</ul>



<h2 class="wp-block-heading">How to decide between Cloudflare R2 vs Amazon S3</h2>



<p class="wp-block-paragraph">Four levers move the outcome. Everything else is noise.</p>



<h3 class="wp-block-heading">Cache hit ratio</h3>



<p class="wp-block-paragraph">The dominant term on both platforms, and the one people skip. At a high hit ratio the origin barely gets touched and both bills collapse toward the storage line. At a low hit ratio, R2 bills read operations and S3 bills origin GETs, and both climb with traffic. Media should be trivially cacheable because the files never change. If yours is not caching, that is a configuration problem worth fixing before you pick a vendor, because fixing it changes the answer.</p>



<h3 class="wp-block-heading">Object count, not library size</h3>



<p class="wp-block-paragraph">WordPress does not store one file per upload. It stores the original plus every registered image size, and themes and page builders happily register more. Ten thousand uploads can easily be sixty thousand objects, each one a write on migration and a read on a cache miss. Count them before you model anything:</p>



<pre class="wp-block-code"><code>cd /path/to/wordpress

# Objects you are about to create in the bucket
find wp-content/uploads -type f | wc -l

# Bytes
du -sh wp-content/uploads

# How much of that count is derivative sizes rather than originals
find wp-content/uploads -type f -regextype posix-extended 
  -regex '.*-[0-9]+x[0-9]+.(jpe?g|png|webp|avif)$' | wc -l</code></pre>



<p class="wp-block-paragraph">That last number is usually the surprise. If most of your object count is thumbnails, trimming unused registered sizes before migrating is the cheapest optimization available, and it shrinks writes, reads, and storage at once.</p>



<h3 class="wp-block-heading">How cold the library actually is</h3>



<p class="wp-block-paragraph">Most libraries are a long tail. A small set of recent files takes nearly all the traffic while the rest sits untouched for years. If that tail is large, S3 has somewhere genuinely cheap to put it and R2 does not. Be careful with R2&#8217;s Infrequent Access class here: it lowers the storage rate but raises both operation rates, adds a retrieval fee, and enforces a minimum duration whether you keep the object or not. It fits write-once, read-almost-never data, not a library anything still links to.</p>



<h3 class="wp-block-heading">What the rest of your stack already is</h3>



<p class="wp-block-paragraph">If your application already runs in AWS and generates derivatives with Lambda on upload, moving to R2 means rebuilding that pipeline against Workers or R2 event notifications. That is real work you do not get paid for. If your site is a VPS from somewhere like InterServer or Contabo with Cloudflare already in front of it, R2 is nearly free work: the DNS is there, the cache is there, and the bucket slots in behind a subdomain you already control.</p>



<h2 class="wp-block-heading">Setup details that change the answer</h2>



<p class="wp-block-paragraph">A few configuration choices matter more than the vendor choice. Get these wrong and the cheaper platform produces the bigger bill.</p>



<h3 class="wp-block-heading">Serve from a custom domain, always</h3>



<p class="wp-block-paragraph">On R2 this is the difference between a cached, protected asset host and a throttled test endpoint. Attach one from the dashboard or from Wrangler:</p>



<pre class="wp-block-code"><code>npx wrangler r2 bucket domain add my-media-bucket 
  --domain=cdn.example.com 
  --zone-id=&lt;YOUR_ZONE_ID&gt;

npx wrangler r2 bucket domain list my-media-bucket</code></pre>



<p class="wp-block-paragraph">Then disable public access on the <code>r2.dev</code> subdomain, or you have left a second uncached, unprotected door into the same objects.</p>



<h3 class="wp-block-heading">Verify the cache instead of assuming it</h3>



<p class="wp-block-paragraph">Assumptions about caching are where the money leaks. Ask the edge directly: request the same object twice and read the headers.</p>



<pre class="wp-block-code"><code># Cloudflare in front of R2
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'cf-cache-status|cache-control|age'

# CloudFront in front of S3
curl -sI https://cdn.example.com/2024/07/photo-1024x768.jpg 
  | grep -iE 'x-cache|cache-control|age'</code></pre>



<p class="wp-block-paragraph">First request, expect a miss. Second, expect a hit. If the second still reports a miss, or reports a status meaning the response was never eligible for caching, every image view is costing you an origin read. Fix that before you compare anything.</p>



<h3 class="wp-block-heading">Long cache lifetimes and versioned filenames</h3>



<p class="wp-block-paragraph">Uploaded media is immutable in practice. Nobody edits the bytes of <code>photo-1024x768.jpg</code>, they upload a new file. So cache lifetimes should be long and the object name should change when the content does. If you find yourself purging image caches regularly, the real problem is that your filenames are not versioned.</p>



<p class="wp-block-paragraph">On the Cloudflare side, Smart Tiered Cache is worth enabling for R2 origins. It routes edge misses through an upper tier data center close to your bucket instead of letting every edge location fetch independently, which cuts the number of requests reaching R2 at all.</p>



<h3 class="wp-block-heading">Clean up failed uploads</h3>



<p class="wp-block-paragraph">Large uploads use multipart. Interrupted ones leave orphaned parts that occupy billed storage and do not show up in a normal object listing. Both platforms support lifecycle rules for this, and on R2 you can manage them from Wrangler:</p>



<pre class="wp-block-code"><code>npx wrangler r2 bucket lifecycle list my-media-bucket

npx wrangler r2 bucket lifecycle add my-media-bucket 
  --name=expire-temp 
  --prefix=tmp/ 
  --expire-days=30</code></pre>



<p class="wp-block-paragraph">Configure an abort rule for incomplete multipart uploads on day one, whichever platform you land on. It is the most common source of storage you pay for and cannot see.</p>



<h2 class="wp-block-heading">A decision procedure you can run this afternoon</h2>



<ol class="wp-block-list">
<li><strong>Measure your current cache hit ratio</strong> for image paths, from analytics or a sampled read of access logs. Everything downstream depends on this number.</li>

<li><strong>Count objects, not gigabytes.</strong> Run the <code>find</code> commands above and separate originals from derivative sizes.</li>

<li><strong>Estimate monthly origin reads</strong> as total image requests multiplied by the miss rate. That, not your bandwidth, is what you feed into either pricing calculator.</li>

<li><strong>Treat writes as a one-off migration spike</strong> plus a modest steady rate. Migration is usually the largest write event the bucket ever sees.</li>

<li><strong>Price both against current published rates</strong> on the same day, using Cloudflare&#8217;s R2 calculator and the AWS pricing calculator. Rates move. Do not trust a number copied from a blog post, including this one.</li>

<li><strong>Test with a prefix before committing.</strong> Offload one year of uploads, point the site at it, watch cache status headers and the operations dashboard for a week. A week of real traffic beats any spreadsheet.</li>
</ol>



<p class="wp-block-paragraph">If it comes out close, take R2. Not because it is cheaper in that scenario, but because it has fewer meters to reason about, and a simpler operational model is worth something on the day something breaks.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;S3 charges egress, R2 doesn&#8217;t, so R2 always wins&#8221;</h3>



<p class="wp-block-paragraph">Only true against the worst possible S3 setup. Transfer from an S3 bucket to CloudFront is not billed, so a properly built S3 media stack pays CDN delivery rates on cache misses, not the direct-from-bucket rate. Add CloudFront&#8217;s permanent free allowance and a small site pays nothing on either platform. The gap opens at scale and on origin-heavy traffic, not on the first terabyte.</p>



<h3 class="wp-block-heading">&#8220;R2 is a CDN&#8221;</h3>



<p class="wp-block-paragraph">R2 is object storage that can sit behind a CDN you already have. The caching, the WAF, the bot rules all come from the Cloudflare zone, and none of it applies to the development URL. If the bucket is not behind a custom domain on a zone you control, you have storage without delivery.</p>



<h3 class="wp-block-heading">&#8220;S3 compatible means drop-in&#8221;</h3>



<p class="wp-block-paragraph">It means most tools work with a changed endpoint and a region value of <code>auto</code>. It does not mean every operation, header, and checksum behaves identically. Budget an afternoon for fighting an SDK default, and test uploads, multipart uploads, deletes, and signed URLs against the real bucket before cutting over.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll save money moving cold files to Infrequent Access&#8221;</h3>



<p class="wp-block-paragraph">Sometimes. The lower storage rate arrives with higher operation rates, a retrieval fee, and a minimum duration. If files are cold enough to justify that, they are probably cold enough to belong in a real archive tier, which R2 does not have. Model it against your read pattern rather than assuming a cheaper per-gigabyte number is a cheaper bill.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Cloudflare R2 cheaper than S3 for a WordPress media library?</h3>



<p class="wp-block-paragraph">Usually, once traffic is meaningful and the library is served publicly. For a small site behind a well configured CDN, both can land near zero. R2&#8217;s advantage grows with egress volume and with the number of requests that miss cache, which is why measuring hit ratio comes first.</p>



<h3 class="wp-block-heading">Can I use the same WordPress plugin for both?</h3>



<p class="wp-block-paragraph">Several plugins speak the S3 API and support both, including Media Cloud, Advanced Media Offloader, and Next3 Offload. WP Offload Media officially lists Amazon S3, DigitalOcean Spaces, and Google Cloud Storage, so check current provider support before assuming a swap is free. Whichever you pick, confirm it rewrites existing URLs and not only new uploads.</p>



<h3 class="wp-block-heading">How do I migrate an existing S3 bucket to R2 without downtime?</h3>



<p class="wp-block-paragraph">Cloudflare offers two paths. Super Slurper copies everything in bulk. Sippy migrates incrementally, pulling an object from the source the first time it is requested and serving from R2 afterward. Both are free to use, you pay for the operations they perform against R2, and your source bucket may charge you for the reads.</p>



<h3 class="wp-block-heading">Do I still need a CDN in front of R2?</h3>



<p class="wp-block-paragraph">You need a custom domain on a Cloudflare zone, which is what puts the CDN in front of it. That is not optional for production. The development URL is rate limited by design and gets no cache, WAF, or bot management.</p>



<h3 class="wp-block-heading">Can I keep S3 and just put Cloudflare in front of it?</h3>



<p class="wp-block-paragraph">You can, and it does cut origin reads. Be aware that cache misses then pull from S3 across the public internet, billed as ordinary S3 egress, unlike an S3 origin sitting behind CloudFront. Reasonable as an interim step, rarely the cheapest end state.</p>



<h3 class="wp-block-heading">What about Backblaze B2, Wasabi, or DigitalOcean Spaces?</h3>



<p class="wp-block-paragraph">All viable and worth pricing if you are already modelling. B2 competes on raw storage, Wasabi sells predictability, Spaces is convenient when your droplets are already there. The analysis transfers directly: find the meters, find your cache hit ratio, multiply.</p>



<h3 class="wp-block-heading">How do I monitor the bill after migrating?</h3>



<p class="wp-block-paragraph">Watch operations and cache hit ratio, not storage. Storage grows slowly and predictably. Operations track traffic and configuration mistakes, which is where the unpleasant surprises live. A Grafana Cloud dashboard or a scheduled pull from the provider&#8217;s usage API is enough; the point is that somebody looks weekly rather than at invoice time.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you take one thing from this Cloudflare R2 vs Amazon S3 comparison, take this: the vendor choice is second order. What decides your media offloading bill is how many requests reach the bucket, and that is a function of your cache configuration, your object count, and whether you put a real domain in front of the storage.</p>



<p class="wp-block-paragraph">Fix the cache first. Count the objects. Then price both, on the same day, against your own numbers. Most of the time R2 comes out ahead for public media, S3 comes out ahead when the library is entangled with AWS services or needs a genuine archive tier, and the difference is smaller than the internet suggests.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help moving a media library without breaking it?</h2>



<p class="wp-block-paragraph">Media offloading looks like a plugin install and turns into a URL rewriting, caching, and permissions problem. I work with teams on the parts that are easy to get subtly wrong:</p>



<ul class="wp-block-list">
<li>Modelling R2 against S3 using your real object counts, request volumes, and cache hit ratio instead of a generic calculator</li>

<li>Planning and running the migration: bulk or incremental copy, URL rewriting, and a rollback path if the cutover misbehaves</li>

<li>Custom domain, cache rule, and header configuration so origin reads collapse instead of tracking your traffic</li>

<li>Bucket permissions, CORS, signed URL flows for private downloads, and lifecycle rules for orphaned multipart uploads</li>

<li>Trimming registered image sizes and derivative sprawl before migration so you stop paying to store thumbnails nothing links to</li>

<li>Dashboards for operations, cache hit ratio, and storage growth so the bill stops being a monthly surprise</li>
</ul>



<p class="wp-block-paragraph">Send me a <code>curl -I</code> of one of your image URLs and a rough object count, and I can usually tell you quickly whether you have a pricing problem or a caching problem.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cloudflare-r2-vs-amazon-s3-media-offloading/">Cloudflare R2 vs Amazon S3 for Media Offloading: Where the Bill Actually Comes From</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: Where the Money Actually Goes</title>
		<link>https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 31 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Predicate Pushdown]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=342</guid>

					<description><![CDATA[<p>A practical comparison of Parquet, JSON and CSV on cost and query performance, covering how each billing model changes the answer, the mechanics behind columnar savings, the failure modes that cancel them out, and a six-step way to model the migration before you commit to it.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A team converts its event logs from JSON to Parquet over a weekend. The Glue job runs clean, the files land in S3, the dashboards keep working. Monday morning someone opens the Athena bill expecting it to have fallen off a cliff, and it has moved by about a tenth.</p>



<p class="wp-block-paragraph">That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.</p>



<p class="wp-block-paragraph">This is a working comparison of <strong>Parquet vs JSON vs CSV</strong> on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.</p>



<h2 class="wp-block-heading">Your billing model decides how much the format matters</h2>



<p class="wp-block-paragraph">Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.</p>



<ul class="wp-block-list"><li><strong>Bytes scanned from object storage.</strong> Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.</li><li><strong>Logical bytes processed.</strong> BigQuery&#8217;s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.</li><li><strong>Compute time.</strong> Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.</li></ul>



<p class="wp-block-paragraph">The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query <em>external</em> data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.</p>



<h2 class="wp-block-heading">CSV: cheap to produce, expensive to interrogate</h2>



<p class="wp-block-paragraph">CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.</p>



<h3 class="wp-block-heading">Where CSV wins</h3>



<ul class="wp-block-list"><li>A human needs to open it. Nothing else comes close.</li><li>Export from a legacy system that offers no other option.</li><li>Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.</li><li>Streaming appends. You can write a line at a time and stop whenever you like.</li></ul>



<h3 class="wp-block-heading">Where CSV does not</h3>



<p class="wp-block-paragraph">Two problems, and the second is worse than the first.</p>



<p class="wp-block-paragraph">The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the <code>lineitem</code> table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.</p>



<p class="wp-block-paragraph">The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.</p>



<p class="wp-block-paragraph">The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether <code>NA</code> means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.</p>



<h2 class="wp-block-heading">JSON: you pay for the schema on every single row</h2>



<p class="wp-block-paragraph">Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called <code>customer_subscription_status</code> carries those thirty-odd bytes on every row, forever.</p>



<h3 class="wp-block-heading">Where JSON wins</h3>



<ul class="wp-block-list"><li>Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.</li><li>Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.</li><li>Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.</li><li>API and webhook payloads, where it is simply what arrives.</li></ul>



<h3 class="wp-block-heading">Where JSON does not</h3>



<p class="wp-block-paragraph">As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.</p>



<p class="wp-block-paragraph">The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.</p>



<h2 class="wp-block-heading">Parquet: what the file is actually doing</h2>



<p class="wp-block-paragraph">Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.</p>



<p class="wp-block-paragraph">A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:</p>



<ul class="wp-block-list"><li><strong>Column pruning.</strong> The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.</li><li><strong>Predicate pushdown.</strong> If a row group&#8217;s maximum value for <code>event_date</code> falls below your filter&#8217;s lower bound, the entire row group is skipped without being read.</li></ul>



<p class="wp-block-paragraph">Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group&#8217;s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes <code>parquet_bloom_probe</code> so you can check which row groups a given value would eliminate.</p>



<p class="wp-block-paragraph">Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.</p>



<h3 class="wp-block-heading">Where Parquet does not win</h3>



<ul class="wp-block-list"><li><strong>Point lookups.</strong> Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.</li><li><strong>Row-level updates.</strong> Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.</li><li><strong>Streaming writes.</strong> The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.</li><li><strong>Tiny datasets.</strong> Below a few megabytes, footer and row group overhead is not worth paying.</li><li><strong>Human inspection.</strong> It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the savings quietly disappear</h2>



<p class="wp-block-paragraph">Every one of these has cost someone a migration that looked excellent in the design document.</p>



<ul class="wp-block-list"><li><strong>Small files.</strong> Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.</li><li><strong>One enormous row group.</strong> Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.</li><li><strong>Select-everything queries in the reporting layer.</strong> Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.</li><li><strong>No partitioning.</strong> Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.</li><li><strong>Gzipped text.</strong> AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.</li><li><strong>Uppercase file extensions.</strong> A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as <code>.GZ</code>, and treats a file with no extension as uncompressed plain text.</li></ul>



<h2 class="wp-block-heading">How to model the change before you migrate</h2>



<p class="wp-block-paragraph">Do not migrate on faith. This takes an afternoon and answers the question properly.</p>



<ol class="wp-block-list"><li><strong>Find out what you scan today.</strong> Every Athena query execution carries a <code>DataScannedInBytes</code> statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.</li><li><strong>Convert one day of data, not all of it.</strong> A single partition is enough to measure with.</li><li><strong>Look inside the resulting file</strong> before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.</li><li><strong>Re-run the same queries</strong> against the converted partition and compare bytes scanned. Measured, not estimated.</li><li><strong>Price the write side.</strong> Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.</li><li><strong>Check who else reads the files.</strong> One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.</li></ol>



<p class="wp-block-paragraph">For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:</p>



<pre class="wp-block-code"><code>-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);</code></pre>



<p class="wp-block-paragraph">The <code>parquet_metadata</code> function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, <code>pyarrow.parquet.ParquetFile</code> exposes the same footer through its <code>metadata</code> attribute, including per-column statistics and whether a bloom filter offset is present.</p>



<p class="wp-block-paragraph">If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena&#8217;s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:</p>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;</code></pre>



<p class="wp-block-paragraph">Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.</p>



<h2 class="wp-block-heading">How I would decide</h2>



<ul class="wp-block-list"><li><strong>Keep the raw landing zone in whatever arrives.</strong> Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.</li><li><strong>Convert once, at the point where data becomes queryable.</strong> Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.</li><li><strong>Partition before you optimise the format.</strong> If there is budget for exactly one change, partition on the column your queries actually filter on.</li><li><strong>Choose the codec by read frequency.</strong> Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.</li><li><strong>Leave small CSVs alone.</strong> Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.</li><li><strong>Watch the scan bill, not the storage bill.</strong> Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Parquet always cheaper than CSV?</h3>



<p class="wp-block-paragraph">On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.</p>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV in practice?</h3>



<p class="wp-block-paragraph">It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.</p>



<h3 class="wp-block-heading">Does compressing my JSON or CSV give me the same savings?</h3>



<p class="wp-block-paragraph">Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.</p>



<h3 class="wp-block-heading">Should I use Parquet or ORC?</h3>



<p class="wp-block-paragraph">For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.</p>



<h3 class="wp-block-heading">Why did my costs not drop after converting to Parquet?</h3>



<p class="wp-block-paragraph">The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull <code>DataScannedInBytes</code> for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.</p>



<h3 class="wp-block-heading">Is JSON still worth keeping now that Parquet has a Variant type?</h3>



<p class="wp-block-paragraph">As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.</p>



<h3 class="wp-block-heading">What file size should I aim for?</h3>



<p class="wp-block-paragraph">Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.</p>



<p class="wp-block-paragraph">An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your data lake costs?</h2>



<p class="wp-block-paragraph">Most format work starts as a cost problem and ends as a layout problem. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all</li><li>Designing the partition scheme and file sizing before the conversion, so the migration is worth running</li><li>Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines</li><li>Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps</li><li>Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice</li><li>Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format</li></ul>



<p class="wp-block-paragraph">If you want a concrete opinion rather than a general one, send me a query&#8217;s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</title>
		<link>https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Adaptive Bitrate]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[MediaConvert]]></category>
		<category><![CDATA[Normalized Minutes]]></category>
		<category><![CDATA[QVBR]]></category>
		<category><![CDATA[Reserved Transcode Slots]]></category>
		<category><![CDATA[Video Transcoding]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=542</guid>

					<description><![CDATA[<p>One checkbox can reprice every output in a MediaConvert job. A working breakdown of how the per-minute charge is assembled from normalized minutes and per-output billing, why failed jobs are free but still expensive, and how to build retry and recovery that classifies before it retries.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A turnaround complaint lands: the promo team&#8217;s masters take too long to appear in the CMS. Someone opens the job template, finds a setting called accelerated transcoding, ticks it, and moves on. Turnaround improves. Nobody mentions it again.</p>



<p class="wp-block-paragraph">The invoice is where it surfaces, and not only on the big outputs. The 240p ladder rung went up. The audio-only rendition went up. Every output in every job that template touches went up, including the ones that never needed the feature.</p>



<p class="wp-block-paragraph">That is not a billing bug. It is the documented rule working as designed, and it is the most common reason a bill moves without anyone changing how much video goes through it. Understanding <strong>MediaConvert cost per minute</strong> starts with accepting that the unit you pay for is not a job, not a source file, and not a minute of compute.</p>



<p class="wp-block-paragraph">What follows is how that charge is assembled, what happens to it when a job fails, and how to build recovery around AWS Elemental MediaConvert. Rates are deliberately absent; AWS changes them and they vary by region. The mechanism is what catches people.</p>



<h2 class="wp-block-heading">How MediaConvert cost per minute is actually assembled</h2>



<p class="wp-block-paragraph">Three rules stack on top of each other. Miss one and your cost model is wrong by a multiple, not a rounding error.</p>



<h3 class="wp-block-heading">You pay per output, not per job</h3>



<p class="wp-block-paragraph">An eight-rung ABR ladder plus an audio-only rendition is nine billable items, each charged for its own duration. AWS bills the duration of the finished assets, not the transcoding time, in hundredth-of-a-minute increments. A ten-minute source producing nine outputs bills as ninety minutes of output.</p>



<p class="wp-block-paragraph">Ladder design is therefore a cost decision, not only a quality one. The bottom rungs are not cheap because they are small. They cost a full pass of the source duration at whatever multiplier their settings attract, whether anyone streams them or not.</p>



<h3 class="wp-block-heading">Normalized minutes, and why multipliers compound</h3>



<p class="wp-block-paragraph">There is no flat rate per output minute. MediaConvert converts each output into normalized minutes, a unit that scales with the work the encode demanded. Resolution, frame rate, codec and number of encoding passes each carry a multiplier, and those multipliers combine. Single-pass to multi-pass is one. 30fps to 60fps is another. Do both to a UHD rung and you have stacked three.</p>



<p class="wp-block-paragraph">Volume discounts then apply to your accumulated normalized minutes for the month, per region, with basic-tier and professional-tier usage accruing separately toward their own thresholds. Two consequences: splitting a workload across regions splits the accrual and can leave you paying a higher effective rate in both, and under consolidated billing in AWS Organizations usage accrues across accounts, so a per-account model understates the discount you actually get.</p>



<h3 class="wp-block-heading">The job-wide setting that reprices everything</h3>



<p class="wp-block-paragraph">Back to the opening. Each output is assessed for its billing tier separately, but only when the job has no job-wide professional-tier feature enabled. Turn one on, accelerated transcoding being the usual culprit, and every output in that job bills at pro-tier rates. An SD rung that would have billed at basic tier gets repriced because a different output in the same job needed to finish faster.</p>



<p class="wp-block-paragraph">If a subset of content genuinely needs acceleration, split it out. Long, visually complex assets get a template with the feature; routine web renditions get one without. Two templates is less tidy than one, and considerably cheaper than a single checkbox repricing your catalogue.</p>



<h2 class="wp-block-heading">Failed jobs are free, which changes how you should retry</h2>



<p class="wp-block-paragraph">MediaConvert charges only when a job reaches COMPLETED. Jobs ending in ERROR are not billed. That sounds like it removes the urgency from failure handling. It does the opposite.</p>



<p class="wp-block-paragraph">A job that fails on output seven of nine may still have written outputs one through six to S3. Those partial artefacts cost storage, and if a naive retry writes to the same prefix you get a mix of old and new segments that a player will serve quite happily until someone notices the ladder is inconsistent. Free at the transcode line, expensive everywhere downstream.</p>



<p class="wp-block-paragraph">&#8220;Not billed&#8221; also makes blind retries feel harmless, so people write them. A loop against a deterministic failure, a corrupt source or a settings mismatch, will never succeed. It burns concurrency paying jobs need and produces identical alerts until the team ignores the channel. The service already retries some conditions itself and exposes a retry count, so read that field before adding an attempt of your own.</p>



<h2 class="wp-block-heading">The queue model is a throughput control, not a cost control</h2>



<p class="wp-block-paragraph">On-demand queues bill per output minute. How many jobs run at once is governed by a service quota for concurrent jobs per account in the region, a quota per individual queue, and a concurrent-jobs setting on each queue. Creating a second queue does not conjure capacity out of the account; you raise it through Service Quotas. Jobs sitting in SUBMITTED rather than moving to PROGRESSING is the signal you have hit the ceiling.</p>



<p class="wp-block-paragraph">Reserved queues work on a different axis. You buy reserved transcode slots against a twelve-month commitment and pay for that capacity regardless of use. Each slot has its own dedicated compute, so one slot runs a given job just as fast as ten would. More slots means more jobs in parallel, not faster jobs. Three things about the commitment deserve a second read before anyone clicks agree:</p>



<ul class="wp-block-list">
<li>You cannot cancel it, and you cannot decrease the slot count.</li>

<li>Adding capacity does not extend the existing term. It starts a fresh twelve-month commitment for the larger count, from the day you add it.</li>

<li>When the plan expires, the queue persists and keeps accepting jobs. MediaConvert just does not run them.</li>
</ul>



<p class="wp-block-paragraph">That last one is the invisible failure in this whole topic. Your submission code gets a job ID back. The API call succeeded. Nothing errors. The jobs simply never progress, and unless you alarm on queue standby time or on the gap between jobs submitted and completed, the first person to notice is whoever asks why the library stopped updating. Set auto-renew deliberately, either way, and diary the term end date.</p>



<p class="wp-block-paragraph">Queue hopping is the pressure valve. Give a job a wait time and a destination queue, and if it is still waiting when the timer expires it moves. The usual shape is reserved to on-demand during a spike. Billing follows the queue the job actually runs in, so a hopped job becomes on-demand spend.</p>



<pre class="wp-block-code"><code>{
  "Settings": { "OutputGroups": [], "Inputs": [] },
  "HopDestinations": [
    {
      "WaitMinutes": 10,
      "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/ondemandqueue",
      "Priority": 25
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">WaitMinutes accepts 1 to 4320. Priority runs from -50 to 50 and defaults to 0, and the optional priority under HopDestinations lets a hopped job land higher or lower in the destination queue than it sat in the original. Hopping is a cost decision as much as a throughput one: it converts capacity you have already paid for into incremental on-demand spend.</p>



<h2 class="wp-block-heading">The failure families that actually bite</h2>



<p class="wp-block-paragraph">MediaConvert publishes numbered error codes with documented causes. Rather than reproduce the table, group them by what you have to do about them, because the correct response differs sharply between groups.</p>



<h3 class="wp-block-heading">Access failures</h3>



<p class="wp-block-paragraph">The service could not open an input or output location. Causes cluster around IAM permissions on the role the job runs under, cross-account bucket policies when the source lives elsewhere, keys that do not exist, and files whose headers cannot be read. Retry these once with a delay, because uploads still in flight are a real condition. If the second attempt fails identically, it is configuration, and no amount of retrying fixes configuration.</p>



<h3 class="wp-block-heading">Settings failures that only appear on some inputs</h3>



<p class="wp-block-paragraph">The nastiest to diagnose, because the template has been fine for the last thousand assets. An encoder initialisation failure on one rung, triggered by an unusual source aspect ratio or frame rate meeting a fixed constraint in the template, is the classic shape. Never retry these; the job fails identically forever. The fix is usually to loosen the constraint rather than pin it harder, because letting the service derive a value the source cannot satisfy removes the whole class of failure.</p>



<h3 class="wp-block-heading">Warnings, which are not errors and are easy to miss</h3>



<p class="wp-block-paragraph">MediaConvert also returns warning codes for problems that do not stop the job. It completes, you get billed, and something about the output is wrong. Video padding inserted to keep audio and video aligned, black frames carried through from the source, a QVBR score well below the rest of the catalogue: none raise an error, all are visible in metrics. Alert only on ERROR and this entire family is invisible to you.</p>



<h3 class="wp-block-heading">Pinned engine versions that quietly expire</h3>



<p class="wp-block-paragraph">MediaConvert groups its releases into job engine versions so you can hold production on one while validating another. Access is requested through AWS Support rather than open to every account, so check whether it applies to you first. The behaviour worth knowing is expiry: requesting an expired version does not fail. The job runs as though no version had been specified, and the field reporting the version used comes back empty. An invalid version, by contrast, returns a 400 and creates no job. The failure mode of pinning is silence.</p>



<h2 class="wp-block-heading">Building failure recovery that does not cost you twice</h2>



<p class="wp-block-paragraph">The pattern that holds up is event-driven, classified and idempotent. In order:</p>



<ol class="wp-block-list">
<li>Subscribe to job state changes in EventBridge rather than polling. The ERROR event carries the code and message in its detail block.</li>

<li>Route by error code, not by status: one target for retryable access conditions, another for deterministic settings failures that need a human, and a suppression path for anything already seen on that asset.</li>

<li>Write a retry&#8217;s outputs to a new prefix, or clear the old one first. Never let attempt two land beside attempt one&#8217;s partial results.</li>

<li>Send a client request token on job creation. Reusing the same token within a minute of a successful request returns the original job instead of creating a duplicate, which protects you when your orchestration retries a create call whose response it never saw.</li>

<li>Cap attempts and land the remainder in a dead letter queue with the original event attached. A failure nobody can see is a failure nobody fixes.</li>

<li>Carry your own identifiers through user metadata on the job, so the returning event joins back to the asset without a lookup table.</li>
</ol>



<pre class="wp-block-code"><code>{
  "source": ["aws.mediaconvert"],
  "detail-type": ["MediaConvert Job State Change"],
  "detail": {
    "status": ["ERROR"]
  }
}</code></pre>



<p class="wp-block-paragraph">The matching event delivers a detail object containing the queue ARN, job ID, status, an errorCode and an errorMessage. Route on errorCode. That one decision is the difference between a recovery system and a retry loop.</p>



<h2 class="wp-block-heading">Measuring what you spend before the invoice tells you</h2>



<p class="wp-block-paragraph">MediaConvert sends metrics to CloudWatch at the end of every job, several mapping directly onto the billing dimensions. The ones I reach for first:</p>



<ul class="wp-block-list">
<li><code>SDOutputDuration</code>, <code>HDOutputDuration</code>, <code>UHDOutputDuration</code>, <code>8KOutputDuration</code> and <code>AudioOutputDuration</code>, in milliseconds per queue. Summed over a month, that is billable output volume by resolution class, days ahead of Cost Explorer.</li>

<li><code>JobErrorsByCode</code>, counting jobs that failed with a particular code. Build the dashboard on this one; it shows which failure family is growing.</li>

<li><code>StandbyTime</code>, how long a job waits before processing starts. Alarm on it and you catch backlogs, and you catch an expired reserved plan.</li>

<li><code>JobsSubmittedCount</code> against <code>JobsCompletedCount</code>. A widening gap is the earliest honest signal something upstream is broken.</li>

<li><code>QVBRAvgQualityLowBitrate</code> and <code>QVBRMinQualityHighBitrate</code>, for quality regressions that never raise an error.</li>
</ul>



<p class="wp-block-paragraph">For attribution rather than volume, enable cost allocation tags and set the billing tags source on your jobs so per-output costs land against a tag you chose. Pushing that into a wider FinOps view through Vantage, CloudZero or a Grafana Cloud dashboard fed from CloudWatch gets you per-title cost without building the aggregation yourself. Either way, these metrics let you model MediaConvert cost per minute continuously instead of discovering it monthly.</p>



<p class="wp-block-paragraph">Transcoding is only part of the line. Storage for sources and renditions in S3, and egress through Amazon CloudFront or whichever CDN fronts delivery, are separate charges and often the larger ones. A cost model that stops at the encoder will be wrong.</p>



<h2 class="wp-block-heading">Troubleshooting a failed job</h2>



<ol class="wp-block-list">
<li>Get the error code first. Everything else follows from which family it belongs to.</li>

<li>Ask whether other assets in the same bucket succeeded in the same window. Yes points at the file, no points at permissions or configuration.</li>

<li>For a settings failure, bisect the job. Strip it to a single video output, confirm it runs, then add outputs and features back one at a time. Tedious, and faster than reasoning about it.</li>

<li>Check warnings on jobs that completed nearby. Warnings often precede errors from the same root cause.</li>

<li>If nothing ran at all, look at the queue before the job: plan status on a reserved queue, standby time on an on-demand one.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Modelling cost from source duration instead of total output duration across the ladder.</li>

<li>Enabling a job-wide professional-tier feature in a shared template and repricing every output in it.</li>

<li>Creating extra on-demand queues for throughput when the account quota is the real ceiling.</li>

<li>Buying reserved slots off a peak-week estimate and paying for idle capacity the other fifty-one.</li>

<li>Retrying every failure identically, including the deterministic ones that cannot succeed.</li>

<li>Alerting on ERROR only, and never looking at warnings or quality scores.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Separate templates by cost profile rather than content type. Anything with a job-wide pro-tier feature gets its own.</li>

<li>Audit the ladder against playback data at least once. Rungs nobody streams are pure cost.</li>

<li>Keep a workload in one region so volume discounts accrue against a single pool.</li>

<li>Run a representative job on an on-demand queue with the reserved simulation setting enabled before committing to slots.</li>

<li>Use job priority instead of extra queues when the goal is for urgent work to jump ahead.</li>

<li>Derive client request tokens from the asset, not from a timestamp, so retries are genuinely idempotent.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I pay for MediaConvert jobs that fail?</h3>



<p class="wp-block-paragraph">No. Billing applies only to jobs reaching COMPLETED status, so a job ending in ERROR is not charged. You may still be paying for partial outputs that reached S3 before the failure, and the storage they occupy until something cleans them up.</p>



<h3 class="wp-block-heading">What is a normalized minute?</h3>



<p class="wp-block-paragraph">The billing unit MediaConvert uses so a minute of demanding output counts for more than a minute of simple output. Resolution, frame rate, codec and encoding passes each contribute a multiplier, and those combine into the normalized total your volume discount tier is measured against.</p>



<h3 class="wp-block-heading">Is a reserved queue cheaper than on demand?</h3>



<p class="wp-block-paragraph">Only if utilisation is genuinely steady. You pay for reserved transcode slots regardless of use, across a twelve-month commitment you cannot cancel or shrink. Bursty catalogues usually do better on demand, where volume discounts apply automatically as monthly usage climbs. Reserved pricing also excludes some features, so confirm everything your templates use is available there before committing.</p>



<h3 class="wp-block-heading">Why are my jobs stuck in SUBMITTED?</h3>



<p class="wp-block-paragraph">On an on-demand queue, it is already running its maximum number of concurrent jobs; raise the quota through Service Quotas rather than adding queues. On a reserved queue, check whether the pricing plan has expired, because an expired queue keeps accepting jobs and never processes them.</p>



<h3 class="wp-block-heading">Does adding queues give me more transcoding capacity?</h3>



<p class="wp-block-paragraph">No. Concurrency is bounded by account-level and queue-level quotas in the region. Extra queues isolate workloads from each other, for instance keeping one client&#8217;s jobs from starving another&#8217;s. Capacity increases come from Service Quotas.</p>



<h3 class="wp-block-heading">How do I stop one setting from repricing my whole job?</h3>



<p class="wp-block-paragraph">Split the job. Job-wide professional-tier features apply their tier to every output, so isolate the assets that need them into their own template and keep routine renditions in a template with no job-wide pro-tier features enabled.</p>



<h3 class="wp-block-heading">Should I run FFmpeg on my own servers instead?</h3>



<p class="wp-block-paragraph">Sometimes. Predictable, modest volume on dedicated hardware from a provider like Contabo or InterServer can undercut per-minute pricing, and you get exact control over encoder flags. What you take on is capacity planning, codec licensing questions, and building the retry and observability machinery the managed service hands you for free. That trade favours MediaConvert for spiky workloads and broadcast-grade features, and self-hosting for steady, simple ladders where an engineer already owns the pipeline.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">MediaConvert cost per minute is not a rate you look up. It is a product: output count, times duration, times a stack of feature multipliers, with a single job-wide setting able to reprice the lot. Get that model into whatever you plan from and the bill stops being a monthly surprise.</p>



<p class="wp-block-paragraph">On the failure side, the codes are documented and the events are structured, so there is no excuse for a retry loop that cannot tell a permissions problem from a settings problem. Classify before you retry. And put an alarm on queue standby time today, because the failure that costs you most is the one where nothing errors at all.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your transcoding pipeline?</h2>



<p class="wp-block-paragraph">I work with teams running video-on-demand pipelines on AWS, usually at the point where the bill or the failure rate has become someone&#8217;s problem. What I actually help with here:</p>



<ul class="wp-block-list">
<li>Auditing job templates and ABR ladders for outputs and multipliers costing more than they return.</li>

<li>Building a cost model from your CloudWatch output-duration metrics so you can forecast rather than react.</li>

<li>Designing EventBridge-driven failure handling that classifies by error code and retries only what can succeed.</li>

<li>Working the reserved versus on-demand decision against your real utilisation, including whether queue hopping earns its keep.</li>

<li>Debugging settings failures that only hit certain sources, and loosening the constraints causing them.</li>

<li>Wiring transcoding, S3 lifecycle and CDN delivery into one cost and reliability picture instead of three.</li>
</ul>



<p class="wp-block-paragraph">Send me a job template, an error code, or a month of output-duration metrics and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</title>
		<link>https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 26 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon EBS]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Explorer]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[NAT Gateway]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Reserved Instances]]></category>
		<category><![CDATA[Savings Plans]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[VPC]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=264</guid>

					<description><![CDATA[<p>Most AWS cost work goes wrong because it starts with the biggest line item, which is also the riskiest. Order the work by blast radius instead: free networking and storage fixes first, performance envelopes one workload at a time, and commitments last. Includes the commands to find the waste and the cuts that look harmless and are not.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/">Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on the third or fourth of the month, and it is never phrased as a question. Someone from finance pastes a screenshot of the AWS bill into a channel, adds a number and a percentage, and waits. Nobody says &#8220;please cause an outage.&#8221; What they say is &#8220;can we get this down.&#8221;</p>



<p class="wp-block-paragraph">That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that &#8220;went fine.&#8221;</p>



<p class="wp-block-paragraph">This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.</p>



<h2 class="wp-block-heading">The cuts that break production are rarely the ones you fear</h2>



<p class="wp-block-paragraph">The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.</p>



<p class="wp-block-paragraph">The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.</p>



<p class="wp-block-paragraph">So the rule I work by: <strong>sequence cost work by blast radius, not by savings size</strong>. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">First, find out what you are actually paying for</h2>



<p class="wp-block-paragraph">The default Cost Explorer view groups by service, which is close to useless here. &#8220;EC2 &#8211; Other&#8221; is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.</p>



<h3 class="wp-block-heading">Group by usage type instead</h3>



<p class="wp-block-paragraph">Usage type names the actual mechanism being billed. Strings like <code>NatGateway-Bytes</code>, <code>NatGateway-Hours</code>, <code>PublicIPv4:InUseAddress</code> and <code>EBS:VolumeUsage.gp2</code> tell you what to go fix. This pulls last month&#8217;s usage types sorted by cost:</p>



<pre class="wp-block-code"><code>aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30</code></pre>



<p class="wp-block-paragraph">That uses GNU <code>date</code>, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.</p>



<h3 class="wp-block-heading">Billing data cannot tell you where your bytes went</h3>



<p class="wp-block-paragraph">This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.</p>



<p class="wp-block-paragraph">VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier zero: savings production cannot notice</h2>



<p class="wp-block-paragraph">Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.</p>



<h3 class="wp-block-heading">The orphan sweep</h3>



<p class="wp-block-paragraph">AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.</p>



<pre class="wp-block-code"><code># Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text</code></pre>



<p class="wp-block-paragraph">Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.</p>



<h3 class="wp-block-heading">NAT Gateway is usually the largest risk-free win</h3>



<p class="wp-block-paragraph">A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.</p>



<p class="wp-block-paragraph">The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.</p>



<pre class="wp-block-code"><code>aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0</code></pre>



<p class="wp-block-paragraph">A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.</p>



<p class="wp-block-paragraph">Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.</p>



<h3 class="wp-block-heading">Log groups that never expire</h3>



<p class="wp-block-paragraph">CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.</p>



<pre class="wp-block-code"><code># Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30</code></pre>



<p class="wp-block-paragraph">Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.</p>



<h3 class="wp-block-heading">Incomplete multipart uploads</h3>



<p class="wp-block-paragraph">Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:</p>



<pre class="wp-block-code"><code>{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Apply it with <code>aws s3api put-bucket-lifecycle-configuration</code>. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier one: savings that move a performance envelope</h2>



<p class="wp-block-paragraph">Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.</p>



<h3 class="wp-block-heading">gp2 to gp3, and the throughput trap</h3>



<p class="wp-block-paragraph">This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.</p>



<p class="wp-block-paragraph">The trap is throughput, not IOPS. gp3&#8217;s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is &#8220;the nightly job got slower&#8221; rather than an alert.</p>



<p class="wp-block-paragraph">Check <code>VolumeReadBytes</code> and <code>VolumeWriteBytes</code> in CloudWatch first, and provision throughput explicitly where the volume needs it:</p>



<pre class="wp-block-code"><code>aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text</code></pre>



<p class="wp-block-paragraph">Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.</p>



<h3 class="wp-block-heading">Rightsizing without guessing</h3>



<p class="wp-block-paragraph">AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.</p>



<pre class="wp-block-code"><code>aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text</code></pre>



<p class="wp-block-paragraph">My working rules:</p>



<ul class="wp-block-list">
<li>Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.</li>

<li>Change one dimension at a time, family or size, so you know what caused any regression.</li>

<li>Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.</li>

<li>Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.</li>

<li>Write down the metric you expect to move and the threshold at which you revert, before you make the change.</li>
</ul>



<h3 class="wp-block-heading">S3 storage classes and the small-object trap</h3>



<p class="wp-block-paragraph">Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.</p>



<p class="wp-block-paragraph">Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.</p>



<p class="wp-block-paragraph">Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier two: commitments, which break budgets rather than uptime</h2>



<p class="wp-block-paragraph">Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:</p>



<ul class="wp-block-list">
<li>Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.</li>

<li>Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.</li>

<li>Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.</li>

<li>Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.</li>

<li>Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.</li>

<li>Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.</li>
</ul>



<p class="wp-block-paragraph">Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.</p>



<p class="wp-block-paragraph">One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill went down and something broke</h2>



<p class="wp-block-paragraph">The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:</p>



<ul class="wp-block-list">
<li><strong>Latency got worse but CPU is fine.</strong> Look at storage. <code>VolumeQueueLength</code> climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling <code>BurstBalance</code> is the tell, and it usually starts hours before anything is user-visible.</li>

<li><strong>Intermittent timeouts after rightsizing.</strong> Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.</li>

<li><strong>Something stopped reaching a service.</strong> If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint&#8217;s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.</li>

<li><strong>Objects are slow or missing from an application path.</strong> Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.</li>
</ul>



<p class="wp-block-paragraph">Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Starting with the biggest number.</strong> Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.</li>

<li><strong>Committing before rightsizing.</strong> This locks in the exact waste you were trying to remove, for years.</li>

<li><strong>Accepting gp3 defaults on high-throughput volumes.</strong> Cheaper storage, halved throughput ceiling, no alert.</li>

<li><strong>Enabling Intelligent-Tiering on buckets full of tiny objects.</strong> Monitoring fees on objects that will never move a tier.</li>

<li><strong>Turning off monitoring to save on monitoring.</strong> Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.</li>

<li><strong>Doing it once.</strong> Cost drifts back within two quarters unless something structural stops it.</li>
</ul>



<h2 class="wp-block-heading">Best practices that make the savings stick</h2>



<ul class="wp-block-list">
<li><strong>Make cost visible before it is a crisis.</strong> AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.</li>

<li><strong>Tag at creation, in the IaC.</strong> Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.</li>

<li><strong>Put non-production on a schedule.</strong> Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.</li>

<li><strong>Make the cheap thing the default in the module.</strong> gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.</li>

<li><strong>Review commitment coverage on a schedule.</strong> Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the fastest way to cut AWS costs without breaking production?</h3>



<p class="wp-block-paragraph">Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.</p>



<h3 class="wp-block-heading">Why is my &#8220;EC2 &#8211; Other&#8221; line item so large?</h3>



<p class="wp-block-paragraph">It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.</p>



<h3 class="wp-block-heading">Is migrating from gp2 to gp3 always cheaper?</h3>



<p class="wp-block-paragraph">No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.</p>



<h3 class="wp-block-heading">Can I cancel a Savings Plan if my usage drops?</h3>



<p class="wp-block-paragraph">Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.</p>



<h3 class="wp-block-heading">Should I use Spot Instances to cut costs?</h3>



<p class="wp-block-paragraph">For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.</p>



<h3 class="wp-block-heading">How do I find out where my NAT Gateway traffic is going?</h3>



<p class="wp-block-paragraph">Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.</p>



<p class="wp-block-paragraph">Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say &#8220;not yet&#8221; when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your AWS bill?</h2>



<p class="wp-block-paragraph">This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:</p>



<ul class="wp-block-list">
<li>A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.</li>

<li>NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.</li>

<li>EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.</li>

<li>CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.</li>

<li>Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.</li>

<li>Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cut-aws-costs-without-breaking-production/">Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</title>
		<link>https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 21 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Hosting & Infrastructure]]></category>
		<category><![CDATA[Amazon EC2]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Fargate]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Containers]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Lightsail]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[Managed Hosting]]></category>
		<category><![CDATA[NAT Gateway]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=262</guid>

					<description><![CDATA[<p>Choosing between Lightsail, EC2 and ECS for a small client workload is not a performance question. It is a question about which constraints you are accepting and how expensive they are to reverse. Profiles of all three with where each wins and loses, the cost levers that actually move the bill, a six-step decision procedure, and the arguments that fall apart on contact with a real project.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;site feels slow, can you take a look.&#8221; What it turned into was a migration. The client&#8217;s Lightsail instance had outgrown its bundle, the plan was to move to EC2, and Lightsail ships an export-to-EC2 feature that makes that sound like a button press.</p>



<p class="wp-block-paragraph">The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.</p>



<p class="wp-block-paragraph">That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.</p>



<p class="wp-block-paragraph">This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn&#8217;t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.</p>



<h2 class="wp-block-heading">Why the exit path matters more than the launch</h2>



<p class="wp-block-paragraph">All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.</p>



<p class="wp-block-paragraph">Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which is cheapest for a small site.&#8221; It is &#8220;how much of this stack is portable, and who operates it after handover.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Amazon Lightsail: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.</li>

<li>The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.</li>

<li>Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.</li>

<li>IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the <em>default</em> VPC in the same account and the same region. Not a custom VPC. Not another account.</li>

<li>Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.</li>

<li>The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.</li>

<li>No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.</li>
</ul>



<p class="wp-block-paragraph">Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail&#8217;s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.</p>



<h2 class="wp-block-heading">Amazon EC2: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.</li>

<li>Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.</li>

<li>Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.</li>

<li>Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.</li>

<li>Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.</li>

<li>Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.</li>

<li>You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.</li>
</ul>



<p class="wp-block-paragraph">The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.</p>



<h2 class="wp-block-heading">Amazon ECS: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.</p>



<ul class="wp-block-list">
<li><strong>ECS on Fargate</strong> bills per vCPU-hour and GB-hour of the resources you <em>declare in the task definition</em>, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.</li>

<li><strong>ECS on EC2</strong> means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.</li>
</ul>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li>The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.</li>

<li>Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.</li>

<li>Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.</li>

<li>Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.</li>

<li>If the app is already containerized and CI already builds images, you are most of the way there.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.</li>

<li>Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.</li>

<li>Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.</li>

<li>Handover is harder. A client&#8217;s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.</li>
</ul>



<p class="wp-block-paragraph">The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.</p>



<pre class="wp-block-code"><code># Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"</code></pre>



<p class="wp-block-paragraph">Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.</p>



<h2 class="wp-block-heading">Lightsail vs EC2 vs ECS on the lines that move the bill</h2>



<p class="wp-block-paragraph">Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.</p>



<ul class="wp-block-list">
<li><strong>Egress.</strong> Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.</li>

<li><strong>NAT gateway.</strong> Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.</li>

<li><strong>Load balancing.</strong> Lightsail&#8217;s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.</li>

<li><strong>Public IPv4.</strong> Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.</li>

<li><strong>Log ingestion.</strong> CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.</li>

<li><strong>Idle resources.</strong> Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.</li>
</ul>



<h2 class="wp-block-heading">A decision procedure for Lightsail vs EC2 vs ECS</h2>



<p class="wp-block-paragraph">Run these in order. The first one that gives a clear answer is the answer.</p>



<ol class="wp-block-list">
<li><strong>Does this workload need to talk privately to other AWS resources?</strong> If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.</li>

<li><strong>Who operates this in a year?</strong> If the answer is &#8220;the client&#8217;s office manager&#8221; or &#8220;nobody,&#8221; bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.</li>

<li><strong>Is the app already containerized, with CI that builds images?</strong> If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.</li>

<li><strong>Is the load steady or spiky?</strong> Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.</li>

<li><strong>How much egress?</strong> Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail&#8217;s bundled allowance is a real argument.</li>

<li><strong>What is the audit story?</strong> If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.</li>
</ol>



<p class="wp-block-paragraph">If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works <em>before</em> you need it. The commands are short:</p>



<pre class="wp-block-code"><code># Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records</code></pre>



<p class="wp-block-paragraph">Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.</p>



<p class="wp-block-paragraph">If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:</p>



<pre class="wp-block-code"><code>aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1</code></pre>



<p class="wp-block-paragraph">You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.</p>



<p class="wp-block-paragraph">Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;Lightsail is just EC2 with a nicer console&#8221;</h3>



<p class="wp-block-paragraph">Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.</p>



<h3 class="wp-block-heading">&#8220;Start on Lightsail, migrate to EC2 when you outgrow it&#8221;</h3>



<p class="wp-block-paragraph">Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.</p>



<h3 class="wp-block-heading">&#8220;Fargate is serverless, so it scales to zero and costs nothing when idle&#8221;</h3>



<p class="wp-block-paragraph">The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word &#8220;serverless&#8221; gives you.</p>



<h3 class="wp-block-heading">&#8220;Containers are better practice, so ECS is the professional choice&#8221;</h3>



<p class="wp-block-paragraph">Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.</p>



<h3 class="wp-block-heading">&#8220;Reserved Instances make EC2 cheaper, so always pick EC2&#8221;</h3>



<p class="wp-block-paragraph">Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client&#8217;s behalf is a conversation, not a default.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Lightsail cheaper than EC2 for a small website?</h3>



<p class="wp-block-paragraph">Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.</p>



<h3 class="wp-block-heading">Can a Lightsail instance connect to an RDS database?</h3>



<p class="wp-block-paragraph">Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the <em>default</em> VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.</p>



<h3 class="wp-block-heading">Does everything migrate when I export Lightsail to EC2?</h3>



<p class="wp-block-paragraph">No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel &amp; WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.</p>



<h3 class="wp-block-heading">Is ECS overkill for a single small container?</h3>



<p class="wp-block-paragraph">Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.</p>



<h3 class="wp-block-heading">Why is my Fargate bill higher than the vCPU-hour rate suggests?</h3>



<p class="wp-block-paragraph">Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.</p>



<h3 class="wp-block-heading">Should I use Lightsail container service instead of ECS?</h3>



<p class="wp-block-paragraph">Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.</p>



<p class="wp-block-paragraph">One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.</p>



<p class="wp-block-paragraph">Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion before you commit?</h2>



<p class="wp-block-paragraph">Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:</p>



<ul class="wp-block-list">
<li>Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses</li>

<li>Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency</li>

<li>Planning and executing Lightsail to EC2 migrations, including the parts that do not export</li>

<li>Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it</li>

<li>Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy</li>

<li>Writing the handover runbook so the next person can operate it without calling you</li>
</ul>



<p class="wp-block-paragraph">If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/lightsail-vs-ec2-vs-ecs-small-workloads/">Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</title>
		<link>https://john-nessime.com/blog/devops/reduce-saas-api-costs/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 14 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[API Cost Optimization]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Cache Control]]></category>
		<category><![CDATA[Cache Stampede]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[ETag]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[HTTP Headers]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Polling]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Redis]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Retry Budgets]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Vendor Spend]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=229</guid>

					<description><![CDATA[<p>Third-party API spend is the one production signal with no error rate attached to it, which is why it creeps up quietly. This is a working engineer's guide to reducing SaaS API costs by changing the shape of your calls: reading the billing unit before you optimise anything, killing pointless polling with conditional requests and webhooks, collapsing N+1 patterns, caching with stampede protection and per-tenant keys, stopping your own retry amplification, and attributing spend so you can prove the work paid off.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually comes from finance, not from monitoring. A vendor line on the card went up again, nobody shipped a feature that month, and now somebody wants an explanation by Friday.</p>



<p class="wp-block-paragraph">Here is the awkward part: nothing was broken. No alert fired, no error rate moved, no dashboard turned red. Third-party API spend is one of the few production signals with no failure mode attached to it. A background job that polls a vendor every thirty seconds and gets back the exact same payload it got last time looks, from the outside, identical to a healthy integration. It just costs money every single time.</p>



<p class="wp-block-paragraph">This post is about how to reduce SaaS API costs by changing the shape of your calls rather than by renegotiating your contract. Six levers, in the order I&#8217;d pull them: read the billing unit, kill pointless polling, collapse N+1 patterns, cache honestly, stop your own retry amplification, and attribute spend so you can tell whether any of it worked.</p>



<h2 class="wp-block-heading">Start with the billing unit, not the call count</h2>



<p class="wp-block-paragraph">The single most common mistake I see is optimising HTTP requests when the vendor doesn&#8217;t bill HTTP requests. Before you touch any code, read the pricing page carefully enough to answer one question: <em>what is the countable thing?</em></p>



<p class="wp-block-paragraph">It varies more than people expect:</p>



<ul class="wp-block-list">
<li><strong>Per request.</strong> The obvious one. Geocoding, enrichment, most REST endpoints.</li>

<li><strong>Per session.</strong> Google Maps Platform bills Places Autocomplete this way when you pass a session token: the whole keystroke stream plus the terminating Place Details call is grouped and billed as one unit. Omit the token, or reuse one across sessions, and every keystroke request is billed individually.</li>

<li><strong>Per token issued.</strong> Auth0&#8217;s machine-to-machine quotas count access tokens obtained through the client credentials flow, not the API calls you make with them. A service that fetches a fresh token on every outbound request is burning the meter for no reason.</li>

<li><strong>Per input and output token.</strong> LLM providers. Output is typically priced higher than input, which makes &#8220;be concise&#8221; a cost control and not just a style note.</li>

<li><strong>Per record, event, or message.</strong> Twilio, SendGrid, most streaming and messaging platforms. One API call can carry many billable units.</li>

<li><strong>Not at all.</strong> Stripe doesn&#8217;t bill you per API call; the constraint there is a rate limit, not a line item. Optimising those calls buys you reliability headroom, not money.</li>
</ul>



<p class="wp-block-paragraph">Getting this wrong sends you down the wrong path for weeks. Batching requests against an endpoint that bills per record changes your request count and leaves the invoice exactly where it was. Read the unit first.</p>



<h2 class="wp-block-heading">Lever one: stop polling for things that rarely change</h2>



<p class="wp-block-paragraph">This is almost always the biggest single line, and it&#8217;s invisible because it&#8217;s steady. A sync job on a five-minute cron is 288 calls a day per resource. Multiply by resources, by environments, by the replicas you quietly scaled to three, and you get a number with no relationship at all to how often the data changed.</p>



<h3 class="wp-block-heading">Conditional requests, when the vendor honours them</h3>



<p class="wp-block-paragraph">If the API returns an <code>ETag</code> or <code>Last-Modified</code> header, store it and send it back on the next poll as <code>If-None-Match</code> or <code>If-Modified-Since</code>. If nothing changed you get a <code>304 Not Modified</code> with an empty body.</p>



<pre class="wp-block-code"><code># First call: capture the ETag from the response headers
curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $TOKEN" 
  https://api.github.com/repos/OWNER/REPO/pulls

# Second call: send it back and check the status code only
curl -sS -o /dev/null -w '%{http_code}n' 
  -H "Authorization: Bearer $TOKEN" 
  -H 'If-None-Match: "PASTE_THE_ETAG_HERE"' 
  https://api.github.com/repos/OWNER/REPO/pulls</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> dumps response headers to stdout so you can see the <code>etag</code> value. <code>-w '%{http_code}'</code> on the second call prints just the status, which is all you care about: <code>304</code> means the resource is unchanged and you saved a payload.</p>



<p class="wp-block-paragraph">Whether that saves <em>money</em> depends on the vendor. GitHub documents that a conditional request returning 304 doesn&#8217;t count against your primary rate limit when it&#8217;s correctly authorised. Plenty of other vendors return 304 happily and bill it anyway. Test against your own invoice before you assume.</p>



<p class="wp-block-paragraph">Three things that bite people with ETags:</p>



<ul class="wp-block-list">
<li>They&#8217;re per-page on paginated endpoints. A 304 on page one says nothing about pages two through five, so store one ETag per page.</li>

<li>They can be scoped to the credential. Rotate a token and your stored ETags may stop matching, which quietly turns your conditional requests back into full ones.</li>

<li>GraphQL endpoints generally don&#8217;t do conditional requests at all. You cache those yourself, keyed on a hash of the query and its variables.</li>
</ul>



<h3 class="wp-block-heading">Webhooks, when they exist</h3>



<p class="wp-block-paragraph">The proper fix is to stop asking. If the vendor supports webhooks, the call count collapses from &#8220;one per interval&#8221; to &#8220;one per actual change&#8221;, which is usually a difference of two orders of magnitude on a slow-moving resource.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. Webhooks need a publicly reachable endpoint, signature verification, idempotent handling because vendors redeliver, a dead-letter path, and a reconciliation sweep for deliveries that never arrived. On a resource that changes every few seconds, polling is simpler and the savings are small. On one that changes twice a week, the webhook pays for itself immediately.</p>



<p class="wp-block-paragraph">A middle option people forget: many APIs expose a cheap &#8220;has anything changed&#8221; endpoint, a sync cursor, or a delta token. Poll that on a tight interval and only fetch the expensive payload when it moves.</p>



<h2 class="wp-block-heading">Lever two: collapse the N+1 pattern</h2>



<p class="wp-block-paragraph">You know this one from databases, and it shows up identically against third-party APIs. Fetch a list of 200 items, then fetch each item&#8217;s detail individually: that&#8217;s 201 billable calls where the vendor probably offers a way to do it in one or two. Look for:</p>



<ul class="wp-block-list">
<li><strong>Expansion or embed parameters.</strong> Many REST APIs let you inline related objects in the list response so the detail calls disappear entirely.</li>

<li><strong>Field selection.</strong> Asking for fewer fields is free on per-request billing but matters a lot where the field tier drives the price. Google Places is explicit about this: requesting a single higher-tier field upgrades the whole response to that tier&#8217;s pricing.</li>

<li><strong>Bulk and batch endpoints.</strong> Salesforce, Algolia, most CRMs and search platforms have one. They&#8217;re usually documented in a corner nobody reads.</li>

<li><strong>Larger pages.</strong> If the endpoint accepts a page size of 200 and you&#8217;re defaulting to 25, you&#8217;re paying eight times as many calls to fetch identical data.</li>
</ul>



<p class="wp-block-paragraph">The page-size one is the cheapest win in this entire post and it&#8217;s a one-line change. Check your client library defaults today.</p>



<h2 class="wp-block-heading">Lever three: cache with a boring, honest TTL</h2>



<p class="wp-block-paragraph">Caching third-party responses is the lever everyone reaches for first and gets subtly wrong. Three failure modes matter more than the cache itself.</p>



<p class="wp-block-paragraph"><strong>Stampede.</strong> A popular key expires, fifty concurrent requests all miss, and fifty identical calls go out to the vendor. You just paid fifty times for one refresh, at the worst possible moment.</p>



<p class="wp-block-paragraph"><strong>No negative caching.</strong> A lookup that returns &#8220;not found&#8221; is usually not cached at all, so the same failing lookup hammers the vendor forever. Cache your 404s. Short TTL, but cache them.</p>



<p class="wp-block-paragraph"><strong>Cache key design.</strong> If your key doesn&#8217;t include the credential or tenant, you will eventually serve one customer&#8217;s data to another. This is the one mistake in this post that turns a cost problem into an incident.</p>



<p class="wp-block-paragraph">You don&#8217;t necessarily need application code for this. An Nginx instance in front of the vendor handles all three, and it&#8217;s the approach I reach for first when several services hit the same API:</p>



<pre class="wp-block-code"><code>proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:20m
                 max_size=1g inactive=60m use_temp_path=off;

server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass https://api.vendor.example;
        proxy_set_header Host api.vendor.example;

        proxy_cache api_cache;
        proxy_cache_key "$request_method$request_uri$http_authorization";

        proxy_cache_valid 200 5m;
        proxy_cache_valid 404 30s;

        proxy_cache_lock on;
        proxy_cache_use_stale updating error timeout;
        proxy_cache_background_update on;

        add_header X-Cache-Status $upstream_cache_status;
    }
}</code></pre>



<p class="wp-block-paragraph">Reading that from the top down, because every directive is there for a reason:</p>



<ul class="wp-block-list">
<li><code>proxy_cache_key</code> includes <code>$http_authorization</code>, so responses are isolated per credential. That&#8217;s the tenant-leak fix.</li>

<li><code>proxy_cache_valid 404 30s</code> is the negative caching. Failed lookups stop being free traffic for the vendor.</li>

<li><code>proxy_cache_lock on</code> is the stampede fix. Only the first request for a given key goes upstream; the rest wait for it.</li>

<li><code>proxy_cache_use_stale updating</code> plus <code>proxy_cache_background_update on</code> serves the stale copy while a single background request refreshes it. Latency stays flat and you don&#8217;t pay a thundering herd at every expiry.</li>

<li><code>X-Cache-Status</code> gives you <code>HIT</code>, <code>MISS</code>, <code>STALE</code> or <code>UPDATING</code> in the response so you can actually measure your hit rate instead of guessing at it.</li>
</ul>



<p class="wp-block-paragraph">The trade-off is worth stating plainly: you&#8217;re now serving data that can be five minutes stale, and you own another moving part. Fine for a currency rate or a product catalogue. Not fine for an inventory count on a checkout page. Set the TTL from how wrong the data is allowed to be, never from how much you want to save.</p>



<p class="wp-block-paragraph">Where the caller is distributed, Redis with a short lock key does the same job in application code, and a small always-on VPS from something like InterServer or DigitalOcean is plenty to host either the proxy or the cache. At the edge, Cloudflare Workers with KV lets you cache vendor responses close to users, which helps when the calls originate from browsers rather than your backend.</p>



<h2 class="wp-block-heading">Lever four: stop paying for your own retries</h2>



<p class="wp-block-paragraph">This produces the sudden spike rather than the slow creep, and people miss it because retries feel like resilience.</p>



<p class="wp-block-paragraph">The vendor wobbles. Every client retries. With a fixed delay they retry in lockstep, so the vendor gets a synchronised wave on top of normal load, struggles longer, and generates more retries. AWS&#8217;s architecture write-up on exponential backoff and jitter is the canonical reference, and its central finding is worth repeating: backoff alone still leaves visible clusters of calls. Randomising each client&#8217;s delay is what decorrelates the herd, and in their simulation it cut total call count substantially.</p>



<pre class="wp-block-code"><code>import random, time

BASE = 0.5      # seconds
CAP  = 30.0     # never sleep longer than this

def sleep_before_retry(attempt):
    # capped exponential backoff, then full jitter
    window = min(CAP, BASE * (2 ** attempt))
    time.sleep(random.uniform(0, window))</code></pre>



<p class="wp-block-paragraph">The cap stops a long outage turning into hour-long sleeps. The <code>random.uniform(0, window)</code> is the jitter: two processes that failed at the same instant now wake at different times. Note that full jitter can occasionally produce a near-zero delay, which is fine for transient errors but too aggressive for throttling responses that reset on a per-minute window. For a <code>429</code>, honour the <code>Retry-After</code> header if the vendor sends one, and use a jitter formula with a guaranteed floor if they don&#8217;t.</p>



<p class="wp-block-paragraph">Three more rules that cost nothing to implement:</p>



<ol class="wp-block-list">
<li><strong>Never retry a 4xx except 429.</strong> A 400 or a 422 will fail identically forever. Every one of those retries is a billed call buying you nothing.</li>

<li><strong>Cap retries as a fraction of traffic.</strong> A retry budget (say, retries may not exceed some small percentage of total requests) is the only control that sees the system-wide picture. Per-client backoff can&#8217;t.</li>

<li><strong>Check your layers.</strong> If the SDK retries three times, your HTTP client retries three times, and your job runner retries three times, one logical operation is up to twenty-seven calls. This stacking is extremely common and almost never intentional.</li>
</ol>



<p class="wp-block-paragraph">For writes, retrying safely needs an idempotency key so the vendor deduplicates rather than creating two of whatever you asked for. Stripe&#8217;s <code>Idempotency-Key</code> header is the pattern most others copied. Without it, teams disable retries on writes entirely and then eat manual reconciliation work instead.</p>



<h2 class="wp-block-heading">Lever five: move latency-tolerant work off the synchronous path</h2>



<p class="wp-block-paragraph">Ask a blunt question about each integration: does a human wait on this response?</p>



<p class="wp-block-paragraph">Nightly enrichment, classification, report generation, bulk address validation, embedding backfills. Nobody is waiting. Several vendors, including the major LLM providers, offer asynchronous batch endpoints at a discount to the synchronous rate in exchange for a looser delivery window. Rates move, so check current terms rather than a number you read somewhere, but the mechanism is stable and the discount is usually substantial.</p>



<p class="wp-block-paragraph">Even without a batch endpoint, moving work behind a queue lets you coalesce. Ten events touching the same customer record within a minute become one enrichment call instead of ten. That deduplication window is often worth more than the batch discount.</p>



<h2 class="wp-block-heading">Lever six: attribute the spend or you&#8217;re guessing</h2>



<p class="wp-block-paragraph">Everything above is unfalsifiable until you can answer: which feature, which tenant, which job. The invoice gives you the total and never the cause. What I&#8217;d put in place, in order:</p>



<ol class="wp-block-list">
<li><strong>Route outbound calls through one place.</strong> A shared HTTP client wrapper or the caching proxy above. If every service instantiates its own client, you have no chokepoint and no measurement.</li>

<li><strong>Emit a counter per call</strong> with labels for vendor, endpoint, status class, and cache result. Keep the label set small; per-tenant labels will blow up your metrics cardinality and trade one bill for another.</li>

<li><strong>Build one unit-cost metric.</strong> Vendor spend per active tenant per month. It&#8217;s the only number that survives growth, because raw spend going up during a good quarter is not a problem.</li>

<li><strong>Alert on call rate, not on the bill.</strong> A rate alert fires in minutes. An invoice fires in thirty days. Grafana, Prometheus, Datadog, CloudWatch, whatever you already run is fine; the point is that the alert exists at all.</li>

<li><strong>Track cache hit rate as a first-class SLI.</strong> When someone changes a cache key format and the hit rate silently drops to zero, this is the graph that tells you before the invoice does.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill jumped and nothing shipped</h2>



<p class="wp-block-paragraph">Work down this list. In my experience the cause is usually in the first four.</p>



<ul class="wp-block-list">
<li><strong>Replica count changed.</strong> Autoscaling or a Kubernetes replica bump multiplies every cron-driven poller. Check whether your scheduled jobs have leader election, or whether all five pods are dutifully polling the same endpoint.</li>

<li><strong>Cache hit rate collapsed.</strong> A changed key format, a Redis eviction policy under memory pressure, or a TTL someone dropped &#8220;temporarily&#8221; during debugging. Graph the hit rate over the period, not the current value.</li>

<li><strong>Retry loop against a persistent 4xx.</strong> Look for a flat, high rate of identical failing calls. This has the distinctive signature of high call volume with zero successful responses.</li>

<li><strong>A non-production environment pointed at production credentials.</strong> Staging, a CI pipeline, or an integration test suite running on every commit. Check whether calls are arriving from IPs you don&#8217;t recognise.</li>

<li><strong>Session or token handling regressed.</strong> A refactor that dropped a session token parameter, or a service that stopped caching its auth token and now mints a new one per request.</li>

<li><strong>Legitimate growth in an expensive segment.</strong> A few large customers using a costly feature move the bill without moving call count much. Not a bug, but worth ruling in before you go hunting for one.</li>
</ul>



<p class="wp-block-paragraph">If the vendor exposes per-key usage breakdowns in their dashboard, split your credentials by service before your next incident. Separate API keys per environment and per major consumer turn a whodunnit into a two-minute lookup.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Optimising the endpoint that&#8217;s easy to change rather than the one that dominates the bill. Rank by spend first.</li>

<li>Caching without a per-credential key, and finding out the hard way in a multi-tenant system.</li>

<li>Treating 304 responses as free when your specific vendor bills them anyway.</li>

<li>Setting TTLs from a savings target rather than a staleness tolerance, and finding out during a customer escalation.</li>

<li>Leaving retry logic enabled at three separate layers and calling the result &#8220;defence in depth&#8221;.</li>

<li>Adding per-tenant labels to outbound-call metrics and moving the cost from the API vendor to the observability vendor.</li>

<li>Shipping the optimisation and never checking the next invoice against the prediction.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth keeping</h2>



<ul class="wp-block-list">
<li>One outbound HTTP client wrapper per service, with timeouts, retries, and metrics configured in exactly one place.</li>

<li>Separate API credentials per environment and per major consumer, so usage dashboards mean something.</li>

<li>Cache auth tokens until shortly before expiry. Refresh early, not on every call.</li>

<li>Add jitter to scheduled jobs as well as to retries. Everything set to run at the top of the hour arrives at the vendor as a spike.</li>

<li>Write down the staleness tolerance for each cached resource next to its TTL, so the next person doesn&#8217;t have to reverse-engineer your reasoning.</li>

<li>Re-check the pricing page after any vendor product update. Billing units change more often than endpoints do.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How do I find out which service is driving my API spend?</h3>



<p class="wp-block-paragraph">Start with the vendor&#8217;s own usage dashboard broken down by API key. If all your services share one key, that&#8217;s the first thing to fix. Failing that, log every outbound call at the client wrapper with the endpoint and calling service, and aggregate for a week. A week of honest logs beats a month of theorising.</p>



<h3 class="wp-block-heading">Do ETags actually reduce SaaS API costs, or just help with rate limits?</h3>



<p class="wp-block-paragraph">It depends entirely on the vendor. GitHub documents that an authorised conditional request returning 304 doesn&#8217;t count against the primary rate limit. Others count every request regardless of status. Run a controlled test: a known number of conditional calls over a billing period, then compare against the usage report. Don&#8217;t assume either way.</p>



<h3 class="wp-block-heading">Is caching third-party API responses allowed?</h3>



<p class="wp-block-paragraph">Usually yes, but not always, and some terms of service restrict how long you may store certain data. Mapping and place data is a common example of restricted caching. Check the terms for the specific vendor before you build a permanent store, particularly if you&#8217;re caching anything personally identifiable.</p>



<h3 class="wp-block-heading">How much can caching realistically cut an API bill?</h3>



<p class="wp-block-paragraph">Nobody can give you a number without knowing your traffic. Estimate it yourself: for a candidate endpoint, count distinct request keys versus total requests over a day. That ratio is roughly your ceiling for hit rate. Multiply by the endpoint&#8217;s share of spend. If the answer is a couple of percent, spend your time elsewhere.</p>



<h3 class="wp-block-heading">Should I always replace polling with webhooks?</h3>



<p class="wp-block-paragraph">No. The comparison is change frequency against poll frequency. If the resource changes almost as often as you poll, you save little and take on signature verification, redelivery handling, and a reconciliation job. Webhooks win decisively on slow-changing resources and on anything where you&#8217;re polling more than a handful of objects.</p>



<h3 class="wp-block-heading">Does rate limiting my own users reduce vendor costs?</h3>



<p class="wp-block-paragraph">Only when user actions map directly to vendor calls, such as a search box that fires an upstream request per keystroke. Debouncing input and enforcing per-plan quotas helps a lot there. For background processing driven by your own schedules, user rate limits change nothing.</p>



<h3 class="wp-block-heading">Should I disable retries to save money?</h3>



<p class="wp-block-paragraph">No. Fix them instead. Capped exponential backoff with jitter, no retries on non-429 client errors, a retry budget, and retry logic living at exactly one layer. Disabling retries entirely trades a cost problem for a reliability problem, and reliability problems are more expensive.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most efforts to reduce SaaS API costs fail because they start with code. They should start with two questions: what is the vendor actually counting, and which of my calls returned information I already had?</p>



<p class="wp-block-paragraph">Answer those honestly and the work usually collapses to a handful of unglamorous changes. Raise a page size. Cache a token. Put a lock around a cache refresh. Delete a poller that was watching something that changes twice a week. None of it is clever. All of it shows up on the next invoice, which is the only review that counts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your API spend under control?</h2>



<p class="wp-block-paragraph">This is work I take on regularly. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Auditing outbound API traffic to find which endpoints, jobs and tenants are driving the bill, and ranking them by spend rather than by how obvious they are.</li>

<li>Building a caching layer in front of chatty vendor APIs, with Nginx, Redis or Cloudflare Workers, including stampede protection and per-tenant key isolation.</li>

<li>Replacing polling jobs with webhook receivers or delta-token syncs, with signature verification, idempotent handling and a dead-letter path.</li>

<li>Auditing timeout and retry configuration across services to remove stacked retries and add proper backoff, jitter and retry budgets.</li>

<li>Moving latency-tolerant workloads onto batch or queue-based paths, with deduplication windows that cut call volume before it reaches the vendor.</li>

<li>Setting up call-rate and cache-hit-rate dashboards and alerts in Prometheus, Grafana, Datadog or CloudWatch so the invoice is never the first signal.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a cost graph that&#8217;s climbing, a job you suspect is over-polling, or a client wrapper you&#8217;re not sure about, send it over and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
