<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Data Lake | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/data-lake/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/data-lake/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 14 Nov 2026 07:01:31 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Data Lake | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/data-lake/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 14 Nov 2026 07:01:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=683</guid>

					<description><![CDATA[<p>Managed Iceberg is not a storage upgrade, it is a decision about who owns compaction, snapshot expiry and orphan cleanup. A practical comparison of S3 Tables against Iceberg in general purpose S3 buckets, covering the maintenance you actually get, the Lake Formation trap in the access path, how the two bills are shaped, and a seven-step decision procedure you can run against a single table.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The dashboard that used to load in eight seconds now takes ninety. Nobody changed the query, nobody changed the schema, and the table is barely bigger than it was last quarter. Someone opens a ticket titled &#8220;Athena slow?&#8221; and it sits there, because there is no error to point at.</p>



<p class="wp-block-paragraph">What changed is the file count. A streaming job has been writing small Parquet files every few minutes for months, and each one is another entry the query planner opens before it can plan anything. The data is fine. The layout rotted.</p>



<p class="wp-block-paragraph">That is the problem underneath the whole S3 Tables vs Iceberg on S3 argument. It is not a storage question. It is a question about who owns table maintenance, and what happens when the answer is nobody. This post covers what S3 Tables actually runs for you, where each option wins, why the access path matters more than most comparisons admit, how the two bills are shaped, and a decision procedure you can apply to a specific table.</p>



<h2 class="wp-block-heading">The failure that doesn&#8217;t announce itself</h2>



<p class="wp-block-paragraph">Iceberg on general purpose S3 works beautifully on day one. You write Parquet, commit a snapshot, Athena reads it. Nothing about that first week tells you it needs upkeep. Three things degrade quietly afterward.</p>



<ul class="wp-block-list">
<li><strong>Small files accumulate.</strong> Planning cost scales with how many files the planner considers, not with how much data you actually scan.</li>



<li><strong>Snapshots pile up.</strong> Every write creates a table version, and old snapshots keep their data files alive. Storage grows even when the row count is flat.</li>



<li><strong>Orphan files never get cleaned.</strong> A killed write can leave files no snapshot references. They are invisible to every query and they bill every month forever.</li>
</ul>



<p class="wp-block-paragraph">None of that produces an alert. No threshold crossing, no failed job, no red panel. It is a slope, not a cliff, which is why teams hit the &#8220;Athena slow?&#8221; ticket eighteen months in with nothing to blame.</p>



<p class="wp-block-paragraph">Amazon S3 Tables exists to take those three jobs off you. That is a real pitch. The question is what you hand over to accept it.</p>



<h2 class="wp-block-heading">What S3 Tables actually runs for you</h2>



<p class="wp-block-paragraph">S3 Tables introduces a distinct bucket type, the table bucket, which holds tables as subresources rather than a prefix full of objects you manage. Inside it, S3 runs three maintenance operations, all on by default. They map one to one onto the standard Apache Iceberg maintenance routines, which tells you exactly what the do-it-yourself equivalent costs in effort.</p>



<h3 class="wp-block-heading">Compaction</h3>



<p class="wp-block-paragraph">Compaction merges small objects into larger ones and applies row-level deletes while it does. It is configured per table, with a target file size defaulting to 512MB and settable as low as 64MB. It reads Parquet, Avro and ORC, and writes Parquet by default.</p>



<p class="wp-block-paragraph">Two limits matter more than the defaults. Compaction does not support the <code>Fixed</code> data type, and it does not support brotli or lz4 compression. If your writer uses lz4 because someone benchmarked it years ago, you are paying for a managed service that cannot do the job you bought it for. Check the writer config before you migrate.</p>



<p class="wp-block-paragraph">Also worth internalizing: compaction runs on an automated schedule you do not control. There is no &#8220;compact now, I have a board report in an hour&#8221; button. Between a burst of small writes and the next pass, the table sits in exactly the state you were trying to avoid.</p>



<h3 class="wp-block-heading">Snapshot management, and the trap inside it</h3>



<p class="wp-block-paragraph">Snapshot management expires old table versions. Two per-table values govern it: a minimum number of snapshots to keep, and a maximum snapshot age. Both conditions must be satisfied before anything is removed, so the minimum count wins over the age setting on a quiet table.</p>



<p class="wp-block-paragraph">Here is the part that bites. Snapshot management does not honor retention set as Iceberg table properties. Configure a branch or tag based retention policy, or a retention in the table metadata longer than what you set through the maintenance API, and snapshot management is <em>disabled</em> for that table.</p>



<p class="wp-block-paragraph">Not overridden. Disabled. S3 stops expiring snapshots, you keep paying storage for every version, and nothing tells you. Someone runs an innocuous <code>ALTER TABLE ... SET TBLPROPERTIES</code> to tag a release for reproducibility, and months later the storage line has doubled on a table nobody thought was growing. If you adopt S3 Tables, make &#8220;nobody sets snapshot retention in table properties&#8221; a written rule and audit for it.</p>



<h3 class="wp-block-heading">Unreferenced file removal</h3>



<p class="wp-block-paragraph">This is orphan cleanup. It deletes data and metadata files no longer referenced by Iceberg metadata once they pass a retention window, configured as a number of days for unreferenced files and a number of days for non-current files.</p>



<p class="wp-block-paragraph">The structural detail is scope. Compaction and snapshot management are per table. Unreferenced file removal is configured <em>at the table bucket level only</em>. You cannot give a compliance-sensitive table a one day orphan window and a scratch table ten days in the same bucket. Different retention behavior means different table buckets, and that is a design-time decision.</p>



<p class="wp-block-paragraph">Editing any of this needs the maintenance configuration permissions, and automated runs surface in CloudTrail, which is your only real visibility into whether maintenance is happening.</p>



<pre class="wp-block-code"><code>s3tables:GetTableMaintenanceConfiguration
s3tables:PutTableMaintenanceConfiguration</code></pre>



<p class="wp-block-paragraph">Grant those to whoever owns the table, not to the pipeline role. A pipeline should never be able to switch off its own cleanup.</p>



<h2 class="wp-block-heading">S3 Tables: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> A high-frequency writer with no dedicated data platform engineer. Change data capture, clickstream, IoT telemetry, log tables: anything producing a steady drip of small files that nobody will schedule a compaction job for. AWS also positions table buckets as supporting substantially higher transaction rates than general purpose buckets, which is a vendor claim worth testing against your own write pattern rather than taking from a marketing page.</p>



<p class="wp-block-paragraph">It wins again when you want maintenance to be a property of the storage rather than a job in someone&#8217;s scheduler. A compaction job in Airflow can be deleted, break silently, or lose its owner when a contract ends. Maintenance that lives in the bucket does not.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Table buckets are not general purpose buckets, and the gap is wider than the name suggests.</p>



<ul class="wp-block-list">
<li>Public access policies and presigned URLs are not supported. If any workflow hands out object URLs, that pattern is gone.</li>



<li>Table bucket and table access policies are capped at 20 KB, which sounds generous until you are writing per-tenant conditions.</li>



<li>Replication has edges: replicas are read-only and compaction does not run on them, tables with tags or branches cannot be replicated, and a table upgraded from Iceberg V2 to V3 cannot be replicated afterward.</li>



<li>Your lifecycle rules, inventory reports, bucket-level scripts and storage class habits do not transfer directly.</li>



<li>Compaction timing is not yours. Where layout freshness matters within the hour, the scheduled model is a genuine constraint.</li>
</ul>



<h2 class="wp-block-heading">Iceberg on general purpose S3: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give this option its real case, because the &#8220;unmanaged&#8221; framing is out of date. Iceberg tables registered in the AWS Glue Data Catalog can have automatic table optimization enabled, covering compaction, snapshot retention and orphan file management. The optimizer watches partitions and starts compaction once file counts and sizes cross a threshold. The choice is no longer managed versus manual. It is closer to managed at the storage layer versus managed at the catalog layer.</p>



<p class="wp-block-paragraph"><strong>Where it wins.</strong></p>



<ul class="wp-block-list">
<li><strong>You control timing.</strong> Compact before a heavy reporting window or after a backfill, because you call the procedure.</li>



<li><strong>One bucket type, one mental model.</strong> Lifecycle rules, storage classes, bucket policies and VPC endpoint patterns keep working.</li>



<li><strong>Cheaper base storage rate.</strong> Table bucket storage carries a premium, and at volume that is not a rounding error.</li>



<li><strong>Exit is cheap.</strong> Iceberg and Parquet in a bucket you own. Moving catalogs is a metadata operation, not a data migration.</li>



<li><strong>Mixed workloads.</strong> Raw drops, partner exports and Iceberg tables can share one storage architecture.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> You own the failure mode. If nobody enables the optimizer or schedules the procedures, nothing runs and nothing complains. The work is not complicated, which is exactly the problem: it is easy enough that everyone assumes someone else did it.</p>



<p class="wp-block-paragraph">The manual path is three Iceberg Spark procedures on a schedule you own:</p>



<pre class="wp-block-code"><code>-- consolidate small data files
CALL catalog.system.rewrite_data_files(table =&gt; 'db.events')

-- drop old table versions
CALL catalog.system.expire_snapshots(table =&gt; 'db.events')

-- delete files no snapshot references
CALL catalog.system.remove_orphan_files(table =&gt; 'db.events')</code></pre>



<p class="wp-block-paragraph">Each accepts arguments for targets and retention that you should set deliberately rather than accept by default. Treat <code>remove_orphan_files</code> with care: it compares storage against metadata, and a short retention window can delete files a concurrent write is still committing.</p>



<p class="wp-block-paragraph">The other cost is compute. Those procedures run on Spark, so an EMR cluster or a Glue job, with its own IAM, network path, failure modes and bill. That is the real thing you buy out of with S3 Tables, and it is a bigger line than people estimate.</p>



<h2 class="wp-block-heading">The access path is a bigger difference than the maintenance</h2>



<p class="wp-block-paragraph">This is the part that generates support tickets, and most comparisons skip it. There are two REST catalog paths to an S3 table, with different authorization models.</p>



<ol class="wp-block-list">
<li><strong>Direct, through the S3 Tables Iceberg REST endpoint.</strong> Any Iceberg-compatible client connects. Authorization is IAM only, via identity-based policies or resource-based policies on the table and table bucket. AWS suggests this for self-managed or partner catalogs, or basic read and write against a single table bucket.</li>



<li><strong>Through the AWS Glue Data Catalog integration.</strong> Tables register as Data Catalog objects, visible to Athena, Redshift, EMR, Glue ETL and Firehose, with third-party clients reaching them through the Glue Iceberg REST endpoint. Authorization is IAM <em>and</em> Lake Formation. Both checks must pass.</li>
</ol>



<p class="wp-block-paragraph">That second point is where teams lose an afternoon. A PyIceberg client with a correct IAM policy still gets denied, because integrating with the Data Catalog registered the tables as Lake Formation resources and no Lake Formation grant exists. The IAM policy is not wrong. It is not sufficient. For external engines to receive credentials, Lake Formation also has to permit full table access for external engines, and a data lake administrator has to apply the grants.</p>



<p class="wp-block-paragraph">If you already run Lake Formation for fine-grained access control, this is a feature and column-level permissions come along for free. If you do not, you have added a governance service, an administrator role and a second permissions model to a project that was supposed to be about file sizes. Price that in honestly.</p>



<h2 class="wp-block-heading">How the two bills are shaped</h2>



<p class="wp-block-paragraph">Skip the per-gigabyte rates. They change, and the decision does not live there. Look at the shape instead.</p>



<p class="wp-block-paragraph"><strong>S3 Tables bills on four dimensions:</strong> storage, requests, a per-object monitoring fee for every object in the table bucket, and compaction charged both per object and per byte processed. The monitoring fee is the one people miss, because it scales with object <em>count</em>. The same workload that makes compaction expensive makes monitoring expensive, and both are driven by how many small files your writer produces.</p>



<p class="wp-block-paragraph"><strong>Iceberg on general purpose S3 bills on:</strong> storage, requests, and the compute you run maintenance on. That compute is lumpy and easy to underestimate, but it is a knob you control. A weekly compaction window on spot capacity costs very differently from continuous optimization.</p>



<p class="wp-block-paragraph">One lever moves both models: <strong>batch your writes</strong>. Committing every thirty seconds instead of every five minutes multiplies object count by ten, and object count feeds the monitoring fee, the compaction fee, the planning time and the Spark hours alike. Fixing the writer is almost always cheaper than choosing storage around a bad writer.</p>



<p class="wp-block-paragraph">Whichever you pick, make the cost visible. Cost allocation tags on the table bucket, a CloudWatch or Grafana panel on object counts, and CloudTrail on maintenance events will teach you more in a month than any spreadsheet estimate teaches you today.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for S3 Tables vs Iceberg on S3</h2>



<p class="wp-block-paragraph">Work through these for a specific table, not for your platform as a whole. Different tables can legitimately land in different places.</p>



<ol class="wp-block-list">
<li><strong>Count the commits per hour.</strong> A handful of commits a day from a batch job writing large files is not a small file problem, and managed compaction solves nothing. Stay on general purpose S3.</li>



<li><strong>Check the writer&#8217;s compression and types.</strong> lz4 or brotli, or a <code>Fixed</code> column, and S3 Tables compaction will not process those files. Fix the writer or stay put.</li>



<li><strong>Ask who owns maintenance today, by name.</strong> If it is a person who will still be here next year and a job that alerts on failure, do it yourself. If the answer is a shrug, that shrug is what you are paying to replace.</li>



<li><strong>Check for presigned URLs, public policies, lifecycle rules or large access policies on this data.</strong> Any yes rules out a table bucket for that table.</li>



<li><strong>Decide the governance model before the storage model.</strong> Want Lake Formation fine-grained control? The Glue Data Catalog path is a real benefit. Want IAM-only simplicity? Use direct access and keep Lake Formation out of it.</li>



<li><strong>Confirm nothing sets snapshot retention in table properties.</strong> If your pipeline or dbt project does, change it or accept that managed snapshot expiry will be silently off.</li>



<li><strong>Try the middle option first.</strong> Enable Glue Data Catalog table optimization on the existing table. Reversible, no data movement, and it answers &#8220;is automated maintenance enough&#8221; without committing you to a new bucket type.</li>
</ol>



<p class="wp-block-paragraph">That last step is the one I reach for first in almost every case. It delivers the outcome you actually want, maintenance that runs without a human, at the cost of a checkbox rather than a migration.</p>



<h2 class="wp-block-heading">Where teams get this wrong</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a storage migration.</strong> It is an operations and governance decision that happens to involve storage.</li>



<li><strong>Migrating the wrong table.</strong> Teams pick the biggest one. The right candidate is the highest commit frequency, which is often small.</li>



<li><strong>Assuming enabled means running.</strong> Table properties can disable snapshot expiry and unsupported compression can skip compaction. Verify in CloudTrail that operations actually fire.</li>



<li><strong>Forgetting concurrency.</strong> Iceberg uses optimistic concurrency, so compaction and your writes can conflict. Compaction retries on failure; your pipeline needs retry logic too, on both sides of this comparison.</li>



<li><strong>Skipping the exit plan.</strong> The data stays open, the bucket type does not. Write down how you would move back before you move in.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is S3 Tables just Iceberg with extra steps?</h3>



<p class="wp-block-paragraph">No. It is standard Apache Iceberg with maintenance run by the service and a purpose-built bucket type underneath. The table format is identical, so engines and data stay portable. What differs is who executes maintenance and which bucket-level features you give up to get it.</p>



<h3 class="wp-block-heading">Can I get managed maintenance without moving to a table bucket?</h3>



<p class="wp-block-paragraph">Yes. AWS Glue Data Catalog table optimization brings automatic compaction, snapshot retention and orphan file management to Iceberg tables in general purpose buckets. For most teams asking this question, that is the answer they actually wanted.</p>



<h3 class="wp-block-heading">Does compaction break time travel?</h3>



<p class="wp-block-paragraph">Compaction writes a new snapshot, so history survives. Snapshot management is what removes time travel, because expiring a snapshot deletes its metadata and can delete the associated data files. If you rely on travelling to a specific point, set the minimum snapshot count and maximum snapshot age to cover that window deliberately.</p>



<h3 class="wp-block-heading">Why does my PyIceberg client get access denied on S3 Tables?</h3>



<p class="wp-block-paragraph">Usually because you are going through the Glue Data Catalog integration, where both IAM and Lake Formation must authorize the request and only the IAM half is configured. Either add the Lake Formation grants and enable external engine access, or switch to the S3 Tables Iceberg REST endpoint for direct IAM-only access.</p>



<h3 class="wp-block-heading">Is S3 Tables more expensive than regular S3?</h3>



<p class="wp-block-paragraph">The storage rate carries a premium, plus a per-object monitoring fee and compaction charges. Against that, you remove the compute you were spending on maintenance. Whether the total is higher depends almost entirely on object count, so model it from your file counts rather than your terabytes.</p>



<h3 class="wp-block-heading">Can I disable compaction and control it myself?</h3>



<p class="wp-block-paragraph">Yes, compaction can be disabled per table through the maintenance configuration API, which also stops the associated charges. Reasonable if you already have a tuned compaction job, though it undercuts most of the reason to be on a table bucket.</p>



<h3 class="wp-block-heading">Which should a small team pick?</h3>



<p class="wp-block-paragraph">A small team with one high-churn streaming table and no data platform engineer is the clearest fit for S3 Tables. A small team running a nightly batch load into a few large tables should stay on general purpose S3 and turn on Glue table optimization if maintenance is the worry.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables vs Iceberg on S3 is not really a performance comparison, and not really a cost comparison either. Both store the same Parquet under the same open table format, and either performs well when the layout is maintained.</p>



<p class="wp-block-paragraph">What you are choosing is where responsibility for compaction, snapshot expiry and orphan cleanup lives, and how confident you are that it will still be happening in two years when the person who set it up has moved on. Low confidence and a high commit rate means managed Iceberg earns its premium. A real owner, a real schedule and large batch writes means you are paying for a problem you do not have.</p>



<p class="wp-block-paragraph">Either way, check the two silent failures before you ship: table properties that disable snapshot expiry, and a writer whose compression codec compaction cannot read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or fixing a table that already rotted?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for teams running Iceberg on AWS:</p>



<ul class="wp-block-list">
<li>Auditing Iceberg tables for small file growth, snapshot bloat and orphan files, with a before-and-after on planning time and storage</li>



<li>Modelling S3 Tables against your real object counts and commit frequency, so the decision comes from your numbers rather than a pricing page example</li>



<li>Setting up AWS Glue Data Catalog table optimization as the low-risk middle path, including the IAM and network configuration it needs</li>



<li>Untangling Lake Formation and IAM when external Iceberg clients get denied through the Glue REST endpoint</li>



<li>Tuning the writer side, where the actual fix usually lives: commit batching, target file sizes, partition layout and compression choices</li>



<li>Building monitoring that makes maintenance visible, so degradation shows up on a dashboard instead of in a &#8220;why is Athena slow&#8221; ticket</li>
</ul>



<p class="wp-block-paragraph">Send me a table&#8217;s manifest summary, a Glue optimizer config, or the planning stage of a slow query, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Athena Partitioning: Why Your Queries Are Still Slow</title>
		<link>https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 07:01:44 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CTAS]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Predicate Pushdown]]></category>
		<category><![CDATA[Query Performance]]></category>
		<category><![CDATA[Small Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=399</guid>

					<description><![CDATA[<p>Partitioning an Athena table is not the same as getting the benefit of it. A walkthrough of the three failure families behind slow queries on partitioned external data: predicates that never prune, catalog lookups that dominate planning, and file layouts that undo everything. Includes partition projection trade-offs, Glue partition indexes, compaction patterns, and how to tell which problem you have.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/">Athena Partitioning: Why Your Queries Are Still Slow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the daily rollup was timing out. Same query, same workgroup, same table, nobody had touched the SQL in months. The only thing that had changed was that the table kept collecting data.</p>



<p class="wp-block-paragraph">That is the part that confuses people, because the table <em>was</em> partitioned. Someone had done the work. The S3 prefixes had tidy <code>key=value</code> folders, the Glue catalog knew about them, and the query had a <code>WHERE</code> clause on the date. By every checklist on the internet, this table was fine.</p>



<p class="wp-block-paragraph">Athena partitioning fails quietly in a way that other performance problems do not. A query that reads every byte of a five terabyte table does not throw an error. It returns the correct answer. It just takes four minutes instead of eight seconds and bills you for the privilege, and the only place the truth shows up is in the bytes-scanned number that nobody looks at until the finance channel does.</p>



<p class="wp-block-paragraph">This post is about the gap between having partitions and getting the benefit of them. I will go through the three failure families that actually cause slow Athena queries on partitioned external data, how to tell which one you are in, and what to do about each. There is some setup DDL along the way, but the point is the mechanism, not the syntax.</p>



<h2 class="wp-block-heading">How Athena decides what to read</h2>



<p class="wp-block-paragraph">Before any of the failures make sense, it helps to be precise about what happens between pressing run and bytes moving off S3.</p>



<p class="wp-block-paragraph">Athena does not scan a table. It builds a list of S3 objects and reads those. Getting to that list involves three separate reductions, and each one can fail independently of the others:</p>



<ol class="wp-block-list">
<li><strong>Partition resolution.</strong> Athena asks the AWS Glue Data Catalog for the partitions belonging to the table. For a Hive-style table this is a <code>GetPartitions</code> call, and it happens before any filtering of your data.</li>

<li><strong>Partition pruning.</strong> Athena compares your <code>WHERE</code> clause against those partition values and throws away the prefixes that cannot match. Objects under discarded prefixes are never fetched at all.</li>

<li><strong>In-file skipping.</strong> Within the surviving files, columnar formats let Athena read only the columns you asked for, and use row group statistics to skip chunks whose min and max values rule them out.</li>
</ol>



<p class="wp-block-paragraph">Step two is the one people mean when they say partitioning. Step one is the one that quietly becomes the bottleneck on tables with a lot of partitions. Step three is the one that gets wiped out by bad file layout even when the first two worked perfectly.</p>



<p class="wp-block-paragraph">Each reduction only helps if the one above it fired. That is the whole shape of the problem.</p>



<h2 class="wp-block-heading">Failure family one: the filter that never pruned</h2>



<p class="wp-block-paragraph">This is the most common one and the least visible. The table is partitioned correctly. The query filters on the partition column. Athena still reads everything.</p>



<p class="wp-block-paragraph">Pruning happens during planning, before Athena has seen a single row. So the planner has to be able to evaluate your predicate against literal partition values using nothing but the query text. Anything that defers the value until runtime defeats it.</p>



<p class="wp-block-paragraph">The usual culprits:</p>



<ul class="wp-block-list">
<li><strong>A function wrapped around the partition column.</strong> Partition values live in the prefix as text. If you write something that parses or casts them before comparing, the planner is comparing the output of a function it has not run yet, so it keeps every partition.</li>

<li><strong>A filter that arrives through a join.</strong> If the partition value comes from the other side of a join rather than from a literal, planning has no value to prune with. This is the expensive one, because the query looks careful. Joining a small control table that says which day to load is a very natural thing to write and a very reliable way to scan the whole fact table.</li>

<li><strong>A subquery that produces the value.</strong> Same problem, same reason.</li>

<li><strong>Filtering the wrong column.</strong> Tables often carry both an event timestamp inside the data and a partition column derived from it. Filtering the timestamp is correct and returns the right rows. It also prunes nothing.</li>
</ul>



<p class="wp-block-paragraph">You do not have to guess. Run the query through <code>EXPLAIN</code> and look at what the table scan node says it is doing.</p>



<pre class="wp-block-code"><code>EXPLAIN
SELECT status, count(*)
FROM logs.access_events
WHERE dt = '2024-05-01';</code></pre>



<p class="wp-block-paragraph">The plan shows the constraint applied to the source. If your partition column appears there with a bound value, pruning will happen. If the predicate has been pushed into a filter node above the scan instead, it will not, and you are about to read the table.</p>



<p class="wp-block-paragraph">The cheaper check is behavioral: run the query with a narrow filter and note the bytes scanned, then widen the filter by ten times and run it again. If the number barely moves, nothing is pruning. Two runs and you have your answer without reading a plan.</p>



<p class="wp-block-paragraph">The fix for the join and subquery cases is unglamorous: resolve the value first, then substitute it into the query as a literal. In an orchestrated pipeline that means one step that computes the partition to load and a second step that templates it into the SQL. Step Functions, Airflow, or a shell script all do this fine. It feels like a step backwards from a single elegant query. It is the difference between reading one prefix and reading all of them.</p>



<h2 class="wp-block-heading">Failure family two: the partition list is the bottleneck</h2>



<p class="wp-block-paragraph">Now the opposite situation. Pruning works, the query reads a tiny slice, and the thing is still slow. Watch where the time goes and you will find most of it elapses before any data is read.</p>



<p class="wp-block-paragraph">That is partition resolution. Athena has to retrieve the partition metadata from Glue before it can prune it, and on a table with hundreds of thousands of partitions that retrieval is a real remote operation with real latency. AWS documents that Athena can query Glue tables with up to ten million partitions but cannot read more than one million partitions in a single scan, which is a ceiling worth knowing about before you design a scheme that approaches it.</p>



<p class="wp-block-paragraph">You get a strong hint from a query that scans almost nothing and still takes twenty seconds. Planning time is not proportional to data volume. It is proportional to partition count.</p>



<h3 class="wp-block-heading">Option one: Glue partition indexes</h3>



<p class="wp-block-paragraph">Glue lets you create an index on a combination of partition keys, which lets the catalog filter server-side rather than handing Athena the full list. Creating the index alone does nothing for Athena. You also have to flip a table property, and this is the step people miss:</p>



<pre class="wp-block-code"><code>ALTER TABLE logs.access_events
SET TBLPROPERTIES ('partition_filtering.enabled' = 'true');</code></pre>



<p class="wp-block-paragraph">The index makes filtering possible on the Glue side; the property tells Athena to actually use it. Set the same property to <code>false</code> to back the change out without dropping the index, which makes this an easy thing to try and easy to reverse.</p>



<p class="wp-block-paragraph">This is the option I reach for first when the table already exists, already has registered partitions, and is also read by other engines. It changes nothing about how the table works.</p>



<h3 class="wp-block-heading">Option two: partition projection</h3>



<p class="wp-block-paragraph">Partition projection takes a different approach. Instead of asking Glue what partitions exist, you describe the pattern in table properties and Athena computes the partition values and their S3 locations in memory. The <code>GetPartitions</code> call disappears entirely.</p>



<p class="wp-block-paragraph">It also solves a maintenance problem. There is nothing to register when new data lands, so no crawler schedule and no repair command. For time-series data with a predictable layout, this is usually the right answer.</p>



<p class="wp-block-paragraph">A date-partitioned table looks like this:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE logs.access_events (
  request_id string,
  status int,
  path string,
  event_time timestamp
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://my-lake/access-events/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.dt.type' = 'date',
  'projection.dt.format' = 'yyyy-MM-dd',
  'projection.dt.range' = '2021-01-01,NOW',
  'projection.dt.interval' = '1',
  'projection.dt.interval.unit' = 'DAYS'
);</code></pre>



<p class="wp-block-paragraph">The <code>range</code> property accepts relative expressions such as <code>NOW</code>, <code>NOW-3YEARS</code> and <code>NOW+1MONTH</code>, which is what makes the table self-maintaining. The <code>format</code> string follows Java&#8217;s date formatter, and the same format must be used for the range bounds.</p>



<p class="wp-block-paragraph">If your prefixes do not follow the standard <code>column=value</code> layout, which is common for Firehose delivery streams and service logs, add a template that maps partition values onto your actual paths:</p>



<pre class="wp-block-code"><code>'storage.location.template' = 's3://my-lake/access-events/${dt}/'</code></pre>



<p class="wp-block-paragraph">Every partition column must appear in the template, and the path must end in a slash.</p>



<h3 class="wp-block-heading">Where partition projection hurts</h3>



<p class="wp-block-paragraph">Projection is the most oversold feature in this whole area, so here are the edges honestly.</p>



<ul class="wp-block-list">
<li><strong>Out-of-range queries return zero rows, not an error.</strong> Query a date before your configured range starts and Athena completes successfully with an empty result. This is the single nastiest behavior in the feature, because a dashboard that silently goes blank looks like a data problem rather than a configuration problem.</li>

<li><strong>Empty partitions cost you.</strong> Athena projects partitions whether or not they exist in S3. AWS&#8217;s own guidance is that if more than half your projected partitions are empty, you are better off with traditional registered partitions.</li>

<li><strong>Enabling it makes Athena ignore registered partition metadata for that table.</strong> Any hand-registered partition pointing somewhere unusual stops being honored.</li>

<li><strong>It is Athena-only.</strong> Redshift Spectrum, EMR and Athena for Spark reading the same table fall back to standard partition metadata. If your lake is shared, projection fixes one consumer and leaves the rest as they were.</li>

<li><strong><code>SHOW PARTITIONS</code> stops being useful.</strong> Projected partitions are not in the catalog, so the command does not list them. Tooling that introspects partitions will need another approach.</li>

<li><strong>Views do not inherit it.</strong> Projection is configured on tables. Set it on the underlying tables, not on views over them.</li>

<li><strong>Projected dates are generated in UTC.</strong> If your prefixes are written in local time, the boundaries will not line up and you will lose or duplicate a day at the edges.</li>
</ul>



<p class="wp-block-paragraph">The <code>enum</code> type deserves its own warning. It is tempting for things like customer IDs or account codes, but AWS recommends keeping enum projections to a few dozen values, because the whole projection configuration lives inside the Glue table metadata and that metadata has a size ceiling. If you need high-cardinality identifiers as a partition key, the <code>injected</code> type is the right tool, with the caveat that queries fail unless every injected column has a filter, and the <code>IN</code> list is capped at a thousand values.</p>



<h2 class="wp-block-heading">Failure family three: too many partitions and too many files</h2>



<p class="wp-block-paragraph">This is the one people create for themselves while trying to fix the first two. Partitioning felt good, so they partitioned harder. Year, month, day, hour, region, tenant. The prefixes look magnificent. The queries get worse.</p>



<p class="wp-block-paragraph">Two mechanisms are working against you at once.</p>



<p class="wp-block-paragraph"><strong>Per-file overhead.</strong> Every object Athena reads costs a listing entry, an S3 GET, and header parsing before a single useful byte is decoded. On a Parquet file that overhead is relatively fixed, so as file size shrinks it becomes the dominant cost. AWS guidance targets splits of roughly 128 MB, and Parquet&#8217;s default row group size is 128 MB with ORC stripes at 64 MB. Below that, the columnar format is paying its own tax without earning it back. Splitting one hour of data across six partition levels turns a healthy 200 MB file into hundreds of files measured in kilobytes.</p>



<p class="wp-block-paragraph"><strong>S3 request rate.</strong> This is the failure mode that arrives as an actual error rather than slowness. S3 enforces a request rate ceiling per prefix, and a query fanning out across an enormous number of small objects can trip it. AWS explicitly calls out unpartitioned or badly structured data causing S3 exceptions on the GET rate. When your query fails with a throttling error rather than a timeout, look at object count, not data volume.</p>



<p class="wp-block-paragraph">Streaming ingestion is the usual source. A delivery stream buffering every sixty seconds produces well over a thousand objects per day per partition, and none of them are big. The layout is not wrong; the buffering is just not aligned with how the data is read.</p>



<h3 class="wp-block-heading">Compacting without breaking the table</h3>



<p class="wp-block-paragraph">The standard fix is a CTAS query that rewrites a slice of the table into fewer, larger objects. There is a constraint you will hit immediately if you try to do a whole table at once: Athena writes a maximum of 100 unique partition and bucket combinations per query, and exceeding it raises <code>HIVE_TOO_MANY_OPEN_PARTITIONS</code>.</p>



<p class="wp-block-paragraph">So the pattern AWS documents is CTAS to create the table with the first batch, then a series of <code>INSERT INTO</code> statements each staying under the same limit:</p>



<pre class="wp-block-code"><code>CREATE TABLE logs.access_events_compacted
WITH (
  format = 'PARQUET',
  parquet_compression = 'SNAPPY',
  partitioned_by = ARRAY['dt'],
  external_location = 's3://my-lake/access-events-compacted/'
) AS
SELECT request_id, status, path, event_time, dt
FROM logs.access_events
WHERE dt &gt;= '2024-01-01' AND dt &lt; '2024-04-01';</code></pre>



<pre class="wp-block-code"><code>INSERT INTO logs.access_events_compacted
SELECT request_id, status, path, event_time, dt
FROM logs.access_events
WHERE dt &gt;= '2024-04-01' AND dt &lt; '2024-07-01';</code></pre>



<p class="wp-block-paragraph">Two things about this that are easy to get wrong. The partition columns must come last in the select list, in the order given to <code>partitioned_by</code>. And <code>INSERT INTO</code> is not allowed against bucketed tables, so if you are also planning to bucket, you cannot use this incremental pattern for the same table.</p>



<p class="wp-block-paragraph">If compaction becomes a routine job rather than a one-off rescue, that is a strong signal to move the table to Apache Iceberg, where compaction is a maintenance operation on a live table instead of a rebuild-and-swap dance.</p>



<h2 class="wp-block-heading">Choosing partition keys without guessing</h2>



<p class="wp-block-paragraph">Most partitioning arguments are really arguments about granularity, and granularity has an arithmetic answer rather than an opinion. Work it backwards from file size.</p>



<ol class="wp-block-list">
<li><strong>Measure compressed daily volume for the table.</strong> Not raw, compressed, because that is what lands in S3.</li>

<li><strong>Divide by your target file size.</strong> Somewhere in the low hundreds of megabytes is a reasonable target. If a day of data does not comfortably exceed that, do not partition below daily. If a day is many gigabytes, hourly is defensible.</li>

<li><strong>Only add a second dimension if it appears in nearly every query.</strong> A partition key that half your queries ignore multiplies your partition count while pruning nothing for those queries. Region and tenant are the classic examples, useful when every consumer is tenant-scoped and pure overhead when they are not.</li>

<li><strong>Use bucketing for high-cardinality filters instead.</strong> Bucketing on something like a user ID gives Athena a way to skip files within a partition without creating a prefix per value.</li>

<li><strong>Sort within files on your second-most-common filter.</strong> Row group statistics only help skip data when values are clustered. Sorted data makes the min and max in the metadata meaningful; randomly ordered data makes every row group look like it might match.</li>
</ol>



<p class="wp-block-paragraph">On the recurring question of a single <code>dt=2024-05-01</code> column versus separate <code>year</code>, <code>month</code> and <code>day</code> columns: a single ISO-formatted date column is easier to filter correctly, is a clean fit for date-type projection, and makes range queries trivial. Split columns are worth it when you genuinely query at month or year granularity often enough to benefit from pruning at that level. Most tables do not, and the split scheme mostly buys you three chances to forget a predicate instead of one.</p>



<h2 class="wp-block-heading">Troubleshooting Athena partitioning problems</h2>



<p class="wp-block-paragraph">A short decision path for when something is wrong and you do not yet know which family you are in.</p>



<ul class="wp-block-list">
<li><strong>Query returns zero rows, no error.</strong> If projection is enabled, check the configured range first. Out-of-range values return empty rather than failing. Also check the format string against the actual prefix text; one is dashed and the other is not more often than you would think.</li>

<li><strong>Bytes scanned barely changes when you narrow the filter.</strong> Pruning is not happening. Look for functions, joins and subqueries on the partition column, then confirm with <code>EXPLAIN</code>.</li>

<li><strong>Query is slow but scans almost nothing.</strong> Planning cost. Consider a Glue partition index with <code>partition_filtering.enabled</code>, or partition projection.</li>

<li><strong>New data is invisible to queries.</strong> Hive-style layouts need <code>MSCK REPAIR TABLE</code>, non-Hive layouts need <code>ALTER TABLE ADD PARTITION</code>, and neither is needed with projection. Confirm which regime the table is actually in before running anything.</li>

<li><strong><code>MSCK REPAIR TABLE</code> runs clean but adds nothing.</strong> It only understands <code>key=value</code> prefixes. It also skips paths containing uppercase characters, which is a genuinely infuriating hour to lose.</li>

<li><strong>Partitions from a different table appear.</strong> <code>MSCK REPAIR TABLE</code> walks subfolders, so a table whose location sits above another table&#8217;s data will adopt its partitions. Give every table its own prefix hierarchy.</li>

<li><strong>Throttling errors from S3.</strong> Object count, not data size. Time to compact.</li>

<li><strong><code>HIVE_TOO_MANY_OPEN_PARTITIONS</code>.</strong> You crossed the write limit in a single CTAS or <code>INSERT INTO</code>. Split it into batches.</li>
</ul>



<p class="wp-block-paragraph">Worth knowing: Athena has no setting that requires queries to filter on a partition column. There is no equivalent of the partition-filter-required flag some engines expose. What you can do is put a bytes-scanned cutoff on the workgroup, so a query that ignores the partition scheme fails fast instead of running to completion.</p>



<pre class="wp-block-code"><code>aws athena create-work-group 
  --name analytics-guarded 
  --configuration '{
    "ResultConfiguration": {"OutputLocation": "s3://my-athena-results/analytics/"},
    "BytesScannedCutoffPerQuery": 10737418240,
    "EnforceWorkGroupConfiguration": true,
    "PublishCloudWatchMetricsEnabled": true
  }'</code></pre>



<p class="wp-block-paragraph">With CloudWatch metrics published per workgroup, bytes scanned becomes a graphable series. Whether you look at it in CloudWatch directly, in Amazon Managed Grafana, or in Grafana Cloud alongside the rest of your telemetry, the point is that a partitioning regression stops being invisible. Cost platforms such as Vantage or CloudZero will attribute the spend back to a workgroup for you if you want the finance-facing version of the same signal.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Partitioning by a column that queries rarely filter on, then wondering why nothing improved.</li>

<li>Adding hourly partitions to a table that receives a few megabytes an hour.</li>

<li>Filtering the timestamp column that lives inside the data instead of the partition column derived from it.</li>

<li>Enabling projection and leaving the range open-ended into the distant past, so most projected partitions point at nothing.</li>

<li>Creating Glue partition indexes and never setting <code>partition_filtering.enabled</code>, so the index sits there doing nothing.</li>

<li>Mixed-case S3 prefixes, which quietly break <code>MSCK REPAIR TABLE</code>.</li>

<li>Treating projection as a drop-in upgrade on a lake that other engines also read.</li>

<li>Compacting into the same prefix the table already points at, which is how you get duplicated rows in the middle of a rewrite.</li>
</ul>



<h2 class="wp-block-heading">Best practices for Athena partitioning</h2>



<ul class="wp-block-list">
<li>Pick granularity from compressed daily volume divided by a target file size, not from instinct.</li>

<li>Use a single ISO date partition column unless you have a concrete reason to split it.</li>

<li>Prefer partition projection for predictable time-series layouts, and Glue partition indexes for shared or irregular tables.</li>

<li>Resolve dynamic partition values in the orchestration layer and pass literals into the SQL.</li>

<li>Store data in Parquet or ORC with a splittable compression codec, and sort within files on a common secondary filter.</li>

<li>Compact streaming output on a schedule, or align the delivery buffer so it produces usable file sizes in the first place.</li>

<li>Put a bytes-scanned cutoff on every workgroup, including the one analysts use interactively.</li>

<li>Track bytes scanned per workgroup as a time series and treat a step change as an incident, not a curiosity.</li>

<li>Test partition changes against a copy first. Rewriting a table&#8217;s layout under live consumers is not something to improvise.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How many partitions is too many for one Athena table?</h3>



<p class="wp-block-paragraph">There is a hard ceiling, since Athena cannot read more than a million partitions in a single scan, but you will feel the pain long before that. The practical limit is where planning time becomes a noticeable fraction of query time. If a highly selective query spends most of its life before reading data, you are over-partitioned regardless of the absolute count.</p>



<h3 class="wp-block-heading">Does partition projection make queries faster on a table with few partitions?</h3>



<p class="wp-block-paragraph">Not meaningfully. Projection removes catalog lookup latency, and on a small table that lookup was never the problem. Its value on small tables is operational rather than performance related, since you stop having to register new partitions at all.</p>



<h3 class="wp-block-heading">Can I use partition projection and registered partitions on the same table?</h3>



<p class="wp-block-paragraph">No. Enabling projection makes Athena ignore the partition metadata registered for that table. It is one regime or the other, which matters if you have hand-registered partitions pointing at locations outside the normal pattern.</p>



<h3 class="wp-block-heading">Why did my Athena query scan the whole table even though I filtered on the partition column?</h3>



<p class="wp-block-paragraph">Almost always because the predicate could not be evaluated at planning time. A function wrapped around the partition column, a value arriving through a join, or a subquery producing the value all defeat pruning. Run <code>EXPLAIN</code> and check whether the constraint reaches the table scan node or sits in a filter above it.</p>



<h3 class="wp-block-heading">Should I partition by year, month and day or by a single date column?</h3>



<p class="wp-block-paragraph">A single ISO-formatted date column is the better default. It maps cleanly onto date-type projection, makes range filters straightforward, and gives you one predicate to remember instead of three. Split columns earn their place only if you routinely query at month or year granularity.</p>



<h3 class="wp-block-heading">Does partitioning help if my data is CSV or JSON?</h3>



<p class="wp-block-paragraph">Yes, because pruning happens at the prefix level and does not care about file format. But you are leaving the larger win on the table. Row-oriented text formats force Athena to read every column of every surviving row, so converting to Parquet or ORC typically cuts far more than partitioning alone. Do both, and do the format conversion first if you have to pick.</p>



<h3 class="wp-block-heading">How do I stop analysts from running unpartitioned queries?</h3>



<p class="wp-block-paragraph">Athena cannot require a partition filter. The workable control is a per-query bytes-scanned cutoff on the workgroup, which cancels a runaway query before it finishes. Pair it with published CloudWatch metrics so you can see who is hitting it and why.</p>



<h3 class="wp-block-heading">Is it worth moving to Iceberg instead of tuning Hive-style partitions?</h3>



<p class="wp-block-paragraph">If you are rewriting files regularly, handling late-arriving data, or fighting the write-side partition limits, Iceberg removes a lot of that work through hidden partitioning and table maintenance operations. If your table is append-only, time-partitioned and stable, Hive-style partitioning with projection is simpler and there is no prize for migrating.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Athena partitioning is not a property of your S3 layout. It is a property of the interaction between your layout, your catalog, and the exact text of your queries, and all three have to cooperate before a single byte is saved. A perfect prefix structure with a predicate the planner cannot evaluate reads the whole table. A perfect predicate against half a million tiny objects reads them all one GET at a time.</p>



<p class="wp-block-paragraph">So measure the thing that tells the truth. Bytes scanned should fall roughly in proportion to how much you narrow the filter. When it does not, one of the three reductions has stopped firing, and now you know which questions to ask to find out which one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Athena tables?</h2>



<p class="wp-block-paragraph">Most of the work here is diagnosis rather than construction. Things I help with:</p>



<ul class="wp-block-list">
<li>Auditing a slow or expensive Athena workload and identifying whether the cost is pruning, planning, or file layout.</li>

<li>Designing a partition scheme from your actual query patterns and daily volumes, including whether to bucket or sort instead.</li>

<li>Migrating tables to partition projection or Glue partition indexes without breaking the other engines reading the same lake.</li>

<li>Building compaction jobs for streaming ingestion that respect the write-side partition limits and do not duplicate data mid-run.</li>

<li>Rewriting orchestrated queries so dynamic partition values resolve to literals and actually prune.</li>

<li>Setting up workgroup guardrails and bytes-scanned monitoring so the next regression shows up on a dashboard rather than an invoice.</li>
</ul>



<p class="wp-block-paragraph">If you send me your table DDL and the <code>EXPLAIN</code> output for one slow query, I can usually tell you which of the three failure families you are in before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/">Athena Partitioning: Why Your Queries Are Still Slow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: What Each Format Actually Costs You</title>
		<link>https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 24 Oct 2026 06:02:47 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Compression]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CSV]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[JSON]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=339</guid>

					<description><![CDATA[<p>Format choice shows up as a bill, not a performance complaint. A working breakdown of where the money actually goes across Parquet, JSON and CSV: column pruning, row-group skipping, the gzip splittability trap, and request overhead. Includes a procedure for measuring the difference on your own data, and an honest account of where Parquet costs you.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;Athena spend tripled last month, nothing changed.&#8221; Nothing had changed on the query side. What changed was that an upstream team started dropping a wider export into the same S3 prefix, still as gzipped CSV, and every dashboard query that touched that table started reading every column of every file to answer a question about two of them.</p>



<p class="wp-block-paragraph">That is the failure mode that makes this comparison worth writing about. The <strong>Parquet vs JSON vs CSV</strong> question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.</p>



<p class="wp-block-paragraph">This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else&#8217;s benchmark, and an honest look at where CSV and JSON still beat the columnar option.</p>



<h2 class="wp-block-heading">The billing model is the whole story</h2>



<p class="wp-block-paragraph">Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.</p>



<p class="wp-block-paragraph">Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery&#8217;s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.</p>



<p class="wp-block-paragraph">That last part catches people. <code>LIMIT 10</code> does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which format is smaller on disk.&#8221; It is &#8220;how few bytes can the engine get away with reading.&#8221; Those diverge fast.</p>



<h2 class="wp-block-heading">Four levers, not one</h2>



<p class="wp-block-paragraph">When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.</p>



<h3 class="wp-block-heading">1. Column pruning</h3>



<p class="wp-block-paragraph">Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.</p>



<h3 class="wp-block-heading">2. Block skipping</h3>



<p class="wp-block-paragraph">Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.</p>



<p class="wp-block-paragraph">The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group&#8217;s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.</p>



<h3 class="wp-block-heading">3. Splittability</h3>



<p class="wp-block-paragraph">This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.</p>



<p class="wp-block-paragraph">Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.</p>



<p class="wp-block-paragraph">If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.</p>



<h3 class="wp-block-heading">4. Request overhead</h3>



<p class="wp-block-paragraph">Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine&#8217;s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">CSV: still the right answer more often than people admit</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.</p>



<p class="wp-block-paragraph">CSV is an excellent transport format and a poor storage format. That distinction is the useful one.</p>



<h2 class="wp-block-heading">JSON: the most expensive way to store a table</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Every record repeats every key name. Store ten million events with a field called <code>customer_identifier</code> and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.</p>



<p class="wp-block-paragraph">JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.</p>



<h2 class="wp-block-heading">Parquet: cheap reads, real obligations</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> This is the part the enthusiastic posts skip.</p>



<ul class="wp-block-list"><li><strong>You cannot append.</strong> Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.</li><li><strong>Schema mismatches across files are a runtime error, not a warning.</strong> Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.</li><li><strong>Timestamps are a compatibility minefield.</strong> The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.</li><li><strong>It is opaque.</strong> You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.</li><li><strong>Writing costs more.</strong> Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.</li></ul>



<p class="wp-block-paragraph">None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.</p>



<h2 class="wp-block-heading">Measure it on your data, in about twenty minutes</h2>



<p class="wp-block-paragraph">Published benchmarks tell you about somebody else&#8217;s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.</p>



<ol class="wp-block-list"><li><strong>Get a real byte count for what you have now.</strong> Not an estimate, the actual prefix.</li></ol>



<pre class="wp-block-code"><code>aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable</code></pre>



<p class="wp-block-paragraph">The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.</p>



<ol start="2" class="wp-block-list"><li><strong>Convert a representative slice, not the whole thing.</strong> One partition is enough. In Athena, a CTAS statement writes the output for you.</li></ol>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';</code></pre>



<p class="wp-block-paragraph">Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.</p>



<ol start="3" class="wp-block-list"><li><strong>Run your three most common queries against both tables and compare bytes scanned.</strong> Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.</li><li><strong>On BigQuery, price it before you run it.</strong> A dry run costs nothing and returns the byte estimate.</li></ol>



<pre class="wp-block-code"><code>bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'</code></pre>



<ol start="5" class="wp-block-list"><li><strong>Inspect the Parquet file you just wrote.</strong> Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.</li></ol>



<pre class="wp-block-code"><code>python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"</code></pre>



<p class="wp-block-paragraph">That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.</p>



<ol start="6" class="wp-block-list"><li><strong>Put a guardrail up before anyone else touches it.</strong> Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.</li></ol>



<p class="wp-block-paragraph">Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;We gzip our CSVs, so we already save the money.&#8221;</strong> Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.</li><li><strong>&#8220;Parquet is always faster.&#8221;</strong> Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.</li><li><strong>&#8220;We&#8217;ll just convert everything to Parquet.&#8221;</strong> Converting a landing zone that receives appended events all day means fighting the format&#8217;s biggest weakness. Land raw, convert on a schedule, query the converted copy.</li><li><strong>&#8220;Storage is cheap so format doesn&#8217;t matter.&#8221;</strong> Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.</li><li><strong>&#8220;Our BI tool caches, so nobody re-scans.&#8221;</strong> Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A short sequence of questions, in order.</p>



<ul class="wp-block-list"><li><strong>Is this data leaving your organisation?</strong> CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.</li><li><strong>Is it arriving continuously, one record at a time?</strong> Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.</li><li><strong>Is it read more than once, by more than one person, with filters?</strong> Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.</li><li><strong>Is the whole dataset smaller than a few hundred megabytes?</strong> Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.</li><li><strong>Are you fighting append semantics, concurrent writers, or schema evolution?</strong> You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.</li></ul>



<p class="wp-block-paragraph">The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV?</h3>



<p class="wp-block-paragraph">It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.</p>



<h3 class="wp-block-heading">Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?</h3>



<p class="wp-block-paragraph">Less than you would think, because once data is loaded into a BigQuery native table it lives in Google&#8217;s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.</p>



<h3 class="wp-block-heading">Is ORC better than Parquet?</h3>



<p class="wp-block-paragraph">They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet&#8217;s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.</p>



<h3 class="wp-block-heading">Why did my query cost the same after I converted to Parquet?</h3>



<p class="wp-block-paragraph">Usually one of three things. You are running <code>SELECT *</code>, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.</p>



<h3 class="wp-block-heading">Should I use Snappy, GZIP or ZSTD for Parquet?</h3>



<p class="wp-block-paragraph">Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.</p>



<h3 class="wp-block-heading">Can I query CSV and Parquet in the same table?</h3>



<p class="wp-block-paragraph">Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.</p>



<h3 class="wp-block-heading">What about the cost of the conversion job itself?</h3>



<p class="wp-block-paragraph">Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.</p>



<p class="wp-block-paragraph">Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help cutting a data lake bill?</h2>



<p class="wp-block-paragraph">Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves</li><li>Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses</li><li>Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control</li><li>Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges</li><li>Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types</li><li>Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end</li></ul>



<p class="wp-block-paragraph">If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>S3 Tables Compaction in Production: The Cost of Convenience</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 22 Oct 2026 06:02:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Query Performance]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Write Amplification]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=773</guid>

					<description><![CDATA[<p>Managed Iceberg compaction in S3 Tables removes real operational work, but it will not tell you when it stops. A practical look at the maintenance jobs that fail silently, the retention cascade that keeps superseded data on your bill for two weeks, how compaction charges actually accumulate, and the handful of knobs worth touching.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The table bucket had been in production a couple of months. Ingest volume was flat, retention said ninety days, and the storage line on the bill kept climbing anyway. Compaction was enabled. The console showed nothing wrong. Someone had already checked twice.</p>



<p class="wp-block-paragraph">The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.</p>



<p class="wp-block-paragraph">That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.</p>



<h2 class="wp-block-heading">What managed compaction actually does</h2>



<p class="wp-block-paragraph">S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.</p>



<p class="wp-block-paragraph">Compaction is the Iceberg <code>rewriteDataFiles</code> routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table&#8217;s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.</p>



<p class="wp-block-paragraph">There are four strategies. <code>auto</code> is the default and picks for you: sort order defined in metadata means <code>sort</code>, otherwise <code>binpack</code>. <code>binpack</code> packs small files into bigger ones. <code>sort</code> orders rows hierarchically by your chosen columns so engines prune more files. <code>z-order</code> interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.</p>



<p class="wp-block-paragraph">Both <code>sort</code> and <code>z-order</code> need a <code>sort_order</code> in your Iceberg table properties and the <code>s3tables:GetTableData</code> permission to set. They also cost more per byte processed than <code>binpack</code>, because they do more work.</p>



<h2 class="wp-block-heading">Failure family one: maintenance that fails without telling anyone</h2>



<p class="wp-block-paragraph">This is the one worth reading the post for. Maintenance jobs have four states: <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, and <code>Not_Yet_Run</code>. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.</p>



<pre class="wp-block-code"><code>aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table</code></pre>



<p class="wp-block-paragraph">The response carries a block for each of <code>icebergCompaction</code>, <code>icebergSnapshotManagement</code>, and <code>icebergUnreferencedFileRemoval</code>, each with its own status, a <code>lastRunTimestamp</code>, and a <code>failureMessage</code>. The three are independent. Compaction reporting <code>Successful</code> tells you nothing about whether the other two are running.</p>



<p class="wp-block-paragraph">The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.</p>



<h3 class="wp-block-heading">Tags and branches disable snapshot expiry for the whole table</h3>



<p class="wp-block-paragraph">If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.</p>



<p class="wp-block-paragraph">Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month&#8217;s bill rather than in their terminal.</p>



<h3 class="wp-block-heading">Iceberg retention properties conflict with the managed config</h3>



<p class="wp-block-paragraph">If <code>history.expire.max-snapshot-age-ms</code> or <code>history.expire.min-snapshots-to-keep</code> is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in <code>metadata.json</code>. It refuses instead.</p>



<p class="wp-block-paragraph">This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.</p>



<pre class="wp-block-code"><code>ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');</code></pre>



<h3 class="wp-block-heading">gc.enabled set to false</h3>



<p class="wp-block-paragraph">Setting the Iceberg <code>gc.enabled</code> table property to <code>false</code> stops both snapshot management and unreferenced file removal. Leave it unset or explicitly <code>true</code>. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.</p>



<h3 class="wp-block-heading">KMS key policy missing the maintenance principal</h3>



<p class="wp-block-paragraph">If the table is encrypted with a customer managed key, that key&#8217;s policy has to grant access to the service principal <code>maintenance.s3tables.amazonaws.com</code>. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.</p>



<h2 class="wp-block-heading">Failure family two: the storage tail you keep paying for</h2>



<p class="wp-block-paragraph">Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.</p>



<p class="wp-block-paragraph">When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from <code>minSnapshotsToKeep</code> (default 1) and <code>maxSnapshotAgeHours</code> (default 120). Both conditions must be satisfied before a snapshot goes.</p>



<p class="wp-block-paragraph">When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for <code>nonCurrentDays</code> (default 10), then are deleted permanently.</p>



<p class="wp-block-paragraph">Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through <code>unreferencedDays</code> (default 3) into the same noncurrent window.</p>



<p class="wp-block-paragraph">Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.</p>



<pre class="wp-block-code"><code># Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'</code></pre>



<p class="wp-block-paragraph">Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.</p>



<h2 class="wp-block-heading">Failure family three: write amplification and the compaction charge</h2>



<p class="wp-block-paragraph">S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.</p>



<p class="wp-block-paragraph">The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.</p>



<p class="wp-block-paragraph">People reach for <code>targetFileSizeMB</code> first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.</p>



<p class="wp-block-paragraph">The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.</p>



<p class="wp-block-paragraph">Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.</p>



<h2 class="wp-block-heading">Failure family four: files compaction quietly skips</h2>



<p class="wp-block-paragraph">Compaction supports Parquet, Avro, and ORC. It does not support the <code>Fixed</code> data type, nor the <code>brotli</code> or <code>lz4</code> compression codecs.</p>



<p class="wp-block-paragraph">That last one matters because <code>lz4</code> is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.</p>



<p class="wp-block-paragraph">Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the <code>write.format.default</code> table property to <code>avro</code> or <code>orc</code>. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.</p>



<h2 class="wp-block-heading">Failure family five: compaction and your writers fighting over the table</h2>



<p class="wp-block-paragraph">Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.</p>



<p class="wp-block-paragraph">When compaction loses you see <code>Failed</code> with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between <code>Failed</code> and <code>Successful</code>.</p>



<p class="wp-block-paragraph">When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.</p>



<h2 class="wp-block-heading">Turning managed compaction off, and when that is right</h2>



<p class="wp-block-paragraph">Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.</p>



<p class="wp-block-paragraph">The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.</p>



<p class="wp-block-paragraph">Disabling is a per-table change, and it does not require zeroing the settings.</p>



<pre class="wp-block-code"><code>aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'</code></pre>



<p class="wp-block-paragraph">From there you run maintenance yourself. Trino exposes an <code>optimize</code> procedure through <code>ALTER TABLE ... EXECUTE</code>, and Spark has the standard Iceberg <code>rewrite_data_files</code> procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.</p>



<p class="wp-block-paragraph">Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting S3 Tables compaction and maintenance</h2>



<p class="wp-block-paragraph">Work through these in order. The first step resolves more cases than the rest combined.</p>



<ol class="wp-block-list">
<li><strong>Check all three job statuses, not one.</strong> Run <code>get-table-maintenance-job-status</code> and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.</li>

<li><strong>Read the failure message literally.</strong> It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means <code>history.expire.*</code> is set.</li>

<li><strong>Status is <code>Disabled</code> or <code>Not_Yet_Run</code> unexpectedly.</strong> Someone replaced the configuration. Confirm with <code>get-table-maintenance-configuration</code>, and remember <code>put-table-maintenance-configuration</code> replaces rather than merges.</li>

<li><strong>Compaction runs but file count stays high.</strong> Check codec and data types against the unsupported list. Files using <code>brotli</code> or <code>lz4</code>, or containing the <code>Fixed</code> type, are outside what compaction touches.</li>

<li><strong>Insufficient-access failures on an encrypted table.</strong> Check the KMS key policy grants the <code>maintenance.s3tables.amazonaws.com</code> service principal, and that the key is enabled and still exists.</li>

<li><strong>Storage still growing with everything green.</strong> Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.</li>
</ol>



<p class="wp-block-paragraph">For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer &#8220;when did this start&#8221; rather than &#8220;what is broken now&#8221;.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the three maintenance jobs as one thing.</strong> They fail independently and for different reasons. &#8220;Maintenance is enabled&#8221; is not a status.</li>

<li><strong>Creating branches or tags on a managed table.</strong> A normal Iceberg workflow that switches off snapshot expiry for the whole table.</li>

<li><strong>Tuning <code>targetFileSizeMB</code> to reduce cost.</strong> It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.</li>

<li><strong>Setting <code>sort</code> or <code>z-order</code> without a sort order defined.</strong> The strategy needs <code>sort_order</code> in table properties and the <code>s3tables:GetTableData</code> permission, and it costs more per byte than <code>binpack</code>.</li>

<li><strong>Running write pipelines with no commit retry.</strong> Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.</li>
</ul>



<h2 class="wp-block-heading">Best practices for running S3 Tables compaction in production</h2>



<ul class="wp-block-list">
<li><strong>Poll job status on a schedule and alert on it.</strong> A small scheduled function calling <code>get-table-maintenance-job-status</code> across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on <code>Failed</code> and on a stale <code>lastRunTimestamp</code>, on whatever dashboard you already watch, Grafana or CloudWatch.</li>

<li><strong>Make maintenance config part of your IaC.</strong> These are settings someone can change with one CLI call and no review. Codify them so drift is visible.</li>

<li><strong>Set retention windows from a real requirement.</strong> Ask how far back anyone needs to time travel, then set <code>maxSnapshotAgeHours</code> and <code>nonCurrentDays</code> to that plus a margin, rather than accepting defaults by omission.</li>

<li><strong>Batch on the write side before tuning anything else.</strong> Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.</li>

<li><strong>Choose the strategy from the query pattern.</strong> One dominant filter column suggests <code>sort</code>. Several equally weighted predicates suggest <code>z-order</code>. Neither suggests leaving it on <code>auto</code>.</li>

<li><strong>Track the compaction billing dimension separately.</strong> It moves independently of storage and it is the earliest signal that a writer&#8217;s behaviour has changed.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I know if S3 Tables compaction is actually running?</h3>



<p class="wp-block-paragraph">Call <code>get-table-maintenance-job-status</code> and read the <code>icebergCompaction</code> block. It returns <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, or <code>Not_Yet_Run</code>, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.</p>



<h3 class="wp-block-heading">Why is my S3 Tables storage growing when compaction says it succeeded?</h3>



<p class="wp-block-paragraph">Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.</p>



<h3 class="wp-block-heading">Can I run my own compaction alongside the managed one?</h3>



<p class="wp-block-paragraph">You can, but it is a poor idea. Both are writers competing under Iceberg&#8217;s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.</p>



<h3 class="wp-block-heading">Does disabling compaction stop all S3 Tables charges?</h3>



<p class="wp-block-paragraph">No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.</p>



<h3 class="wp-block-heading">Is sort or z-order compaction worth the extra cost?</h3>



<p class="wp-block-paragraph">It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour <code>sort</code>. Several dimensions in varying combinations favour <code>z-order</code>. Queries that mostly scan everything justify neither.</p>



<h3 class="wp-block-heading">Can I recover data deleted by S3 Tables maintenance?</h3>



<p class="wp-block-paragraph">Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.</p>



<p class="wp-block-paragraph">What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on <code>get-table-maintenance-job-status</code> that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.</p>



<h2 class="wp-block-heading">Need help with Iceberg tables on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots</li>

<li>Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana</li>

<li>Working out where a bucket&#8217;s storage growth actually comes from, and setting retention windows against a real requirement</li>

<li>Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them</li>

<li>Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick</li>

<li>Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance</li>
</ul>



<p class="wp-block-paragraph">If something is not adding up, send me the output of <code>get-table-maintenance-job-status</code> for the table and a rough description of your write pattern. That is usually enough to say what is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</title>
		<link>https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 06:06:48 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=500</guid>

					<description><![CDATA[<p>S3 holds the documents, Iceberg holds the rows, and everything works right up until a number disagrees with its source PDF. Combining structured and unstructured data on AWS is an identity problem, not a storage one. Here are the five failure families that cause it, why each stays invisible until it isn't, and the ingestion order that avoids most of them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;the contract value in the dashboard is wrong.&#8221; It wasn&#8217;t. The number in the warehouse table was right for the version of the PDF that had been extracted eleven months earlier. Someone had uploaded an amended contract to the same S3 prefix, the extraction job had picked it up, written a new row, and nothing downstream knew which of the two rows was current. Both were true. Only one was correct.</p>



<p class="wp-block-paragraph">That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.</p>



<p class="wp-block-paragraph">The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.</p>



<h2 class="wp-block-heading">What this post covers</h2>



<p class="wp-block-paragraph">Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn&#8217;t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.</p>



<p class="wp-block-paragraph">Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: there is no join key, only a filename</h2>



<p class="wp-block-paragraph">Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.</p>



<p class="wp-block-paragraph">Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.</p>



<h3 class="wp-block-heading">What to use instead</h3>



<p class="wp-block-paragraph">Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.</p>



<p class="wp-block-paragraph">Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for &#8220;this exact bytes-on-disk artefact.&#8221; If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.</p>



<p class="wp-block-paragraph">S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.</p>



<h2 class="wp-block-heading">Failure family two: extraction confidence gets flattened into a fact</h2>



<p class="wp-block-paragraph">Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.</p>



<p class="wp-block-paragraph">Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn&#8217;t go away. You just stopped being able to see it.</p>



<p class="wp-block-paragraph">Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always &#8220;which extractor produced these,&#8221; and without that column the answer is a week of archaeology.</p>



<pre class="wp-block-code"><code>-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);</code></pre>



<p class="wp-block-paragraph">The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.</p>



<h2 class="wp-block-heading">Failure family three: two catalogs, two permission models</h2>



<p class="wp-block-paragraph">This is the one that bites teams who did everything else right.</p>



<p class="wp-block-paragraph">Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.</p>



<p class="wp-block-paragraph">The result is predictable and unpleasant. An analyst who is correctly denied the <code>salary</code> column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.</p>



<p class="wp-block-paragraph">There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:</p>



<ul class="wp-block-list">

<li>Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.</li>


<li>Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.</li>


<li>Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.</li>


<li>Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.</li>

</ul>



<p class="wp-block-paragraph">Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with <code>.metadata.json</code> appended. That file is where classification, tenant, document ID and effective dates belong.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}</code></pre>



<p class="wp-block-paragraph">Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.</p>



<h2 class="wp-block-heading">Failure family four: index decisions you cannot take back</h2>



<p class="wp-block-paragraph">S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.</p>



<p class="wp-block-paragraph">By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.</p>



<p class="wp-block-paragraph">If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.</p>



<pre class="wp-block-code"><code># Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'</code></pre>



<p class="wp-block-paragraph">The dimension has to match your embedding model&#8217;s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.</p>



<p class="wp-block-paragraph">Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing rewrites history</h2>



<p class="wp-block-paragraph">You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter&#8217;s board pack change.</p>



<p class="wp-block-paragraph">This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.</p>



<p class="wp-block-paragraph">Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and &#8220;why did this number move&#8221; has an answer you can query instead of guess at.</p>



<p class="wp-block-paragraph">Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.</p>



<h2 class="wp-block-heading">A shape that works for combining structured and unstructured data on AWS</h2>



<p class="wp-block-paragraph">Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.</p>



<ol class="wp-block-list">

<li><strong>Land raw, enable versioning, enable S3 Metadata.</strong> Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.</li>


<li><strong>Mint the document ID at landing.</strong> Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.</li>


<li><strong>Classify before you extract.</strong> Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.</li>


<li><strong>Extract to a staging table, not straight to the curated table.</strong> Staging gives you somewhere to run validation before anything touches a table an analyst can query.</li>


<li><strong>Merge into Iceberg on document identity, carrying confidence and provenance columns.</strong> Never merge on object key.</li>


<li><strong>Embed separately, keyed by the same document ID.</strong> The vector plane and the table plane are different systems with different refresh rates. Treat them as such.</li>


<li><strong>Join at query time in Athena.</strong> Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.</li>

</ol>



<p class="wp-block-paragraph">Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Athena says the principal has no privilege on the resource</h3>



<p class="wp-block-paragraph">This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.</p>



<h3 class="wp-block-heading">Queries against metadata tables fail with a syntax error</h3>



<p class="wp-block-paragraph">Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.</p>



<h3 class="wp-block-heading">Knowledge base sync fails partway with validation errors</h3>



<p class="wp-block-paragraph">Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.</p>



<h3 class="wp-block-heading">Retrieval returns documents the caller should not see</h3>



<p class="wp-block-paragraph">Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.</p>



<h3 class="wp-block-heading">Athena gets slower week over week on a table that isn&#8217;t growing much</h3>



<p class="wp-block-paragraph">Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Joining on the S3 object key because it was there and it worked in the prototype.</li>


<li>Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.</li>


<li>Applying a confidence threshold and discarding the score.</li>


<li>Creating the vector index before deciding on the embedding model.</li>


<li>Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.</li>


<li>Reprocessing in place, with no record of which extractor produced which value.</li>


<li>Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.</li>


<li>Confidence and provenance columns are not optional. They are the difference between debugging and guessing.</li>


<li>Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.</li>


<li>Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.</li>


<li>Enforce classification in all three planes independently. There is no single choke point.</li>


<li>Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.</li>


<li>Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a lakehouse to combine structured and unstructured data on AWS?</h3>



<p class="wp-block-paragraph">No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.</p>



<h3 class="wp-block-heading">Should extraction output live in the same table as the structured source data?</h3>



<p class="wp-block-paragraph">Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.</p>



<h3 class="wp-block-heading">Is S3 Vectors a replacement for OpenSearch?</h3>



<p class="wp-block-paragraph">For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.</p>



<h3 class="wp-block-heading">How do I keep the vector index in sync with the documents?</h3>



<p class="wp-block-paragraph">Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because &#8220;mostly in sync&#8221; degrades quietly.</p>



<h3 class="wp-block-heading">What breaks first when this architecture is done badly?</h3>



<p class="wp-block-paragraph">Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.</p>



<h3 class="wp-block-heading">Can I use Textract instead of Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.</p>



<h3 class="wp-block-heading">Does Lake Formation govern my documents as well as my tables?</h3>



<p class="wp-block-paragraph">It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.</p>



<p class="wp-block-paragraph">Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help wiring documents and tables together?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:</p>



<ul class="wp-block-list">

<li>Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.</li>


<li>Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.</li>


<li>Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.</li>


<li>Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.</li>


<li>Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.</li>


<li>Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.</li>

</ul>



<p class="wp-block-paragraph">Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Hotel Data Lake on AWS That Agrees With the Night Audit</title>
		<link>https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 24 Sep 2026 06:00:06 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Hospitality Technology]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Booking Systems]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Hotel PMS]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Late-Arriving Data]]></category>
		<category><![CDATA[Medallion Architecture]]></category>
		<category><![CDATA[OPERA Cloud]]></category>
		<category><![CDATA[PCI DSS]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[RevPAR]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Travel Technology]]></category>
		<category><![CDATA[USALI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=461</guid>

					<description><![CDATA[<p>Hotel source data is mutable in the past, so an append-only pipeline drifts away from the PMS without anyone noticing until month close. A practical guide to room-night grain, bitemporal modeling with Apache Iceberg, PMS ingestion, guest data scope, and file physics at hotel volumes.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/">Building a Hotel Data Lake on AWS That Agrees With the Night Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone opens the manager&#8217;s flash report on one screen and the shiny new dashboard on the other. The RevPAR figures don&#8217;t match. Not by a rounding error either, by enough that the revenue meeting stops being about revenue and turns into an argument about which system is lying.</p>



<p class="wp-block-paragraph">The dashboard usually isn&#8217;t lying. It&#8217;s answering a slightly different question, and the cause is almost always the same one: the pipeline recorded what the property management system said on the day it read it, and the PMS has since changed its mind. A reservation got shortened. A rate was corrected. A rebate landed against a folio that closed two weeks ago. The lake never went back to look.</p>



<p class="wp-block-paragraph">This post is about building a <strong>hotel data lake on AWS</strong> that survives that. Not a service tour, because the AWS documentation already covers S3, Glue and Athena better than a blog post can. What it covers instead is the handful of modelling and ingestion decisions that determine whether your numbers reconcile with the night audit six months from now, and what each of those decisions costs you.</p>



<h2 class="wp-block-heading">The failure that shows up three weeks late</h2>



<p class="wp-block-paragraph">Most first attempts at hotel analytics are append-only. Pull yesterday&#8217;s reservations, write Parquet under a folder keyed by the day you pulled it, register the partition, done. Yesterday reconciles perfectly. So does the day after. Confidence goes up.</p>



<p class="wp-block-paragraph">The problem is that hotel source data is mutable in the past, and it stays mutable for a long time. Group blocks get released. Stays get extended at the desk. Adjustments and rebates get posted against nights that already closed. Commission corrections come back from an OTA weeks after checkout. Night audit itself is a rewrite operation: it closes the business date and posts room and tax charges for every occupied room, and if the auditor reruns it or corrects a posting, the figures for that date change after you already stored them.</p>



<p class="wp-block-paragraph">An append-only lake captures a diagonal. For each business date you keep the first thing you ever saw about it, and nothing after. Every individual day looks right on the day you check it. The month-to-date total drifts. Nobody notices until close, and by then you have no idea which nights moved or why.</p>



<p class="wp-block-paragraph">That is the whole reason to be careful about grain and table format. Everything else in a hospitality pipeline is ordinary data engineering.</p>



<h2 class="wp-block-heading">Fix the grain before you fix the pipeline</h2>



<p class="wp-block-paragraph">The core hotel metrics are per-night metrics. Occupancy is rooms sold divided by rooms available. ADR is room revenue divided by rooms sold. RevPAR is room revenue divided by rooms available, which is the same as ADR multiplied by occupancy. All three are defined against a single night.</p>



<p class="wp-block-paragraph">What the PMS hands you is reservations. A reservation spans nights, carries one confirmation number, and may have different rates on different nights of the same stay. If you store reservations and try to compute occupancy from them, you end up writing date-range arithmetic into every query, and two analysts will write it differently.</p>



<h3 class="wp-block-heading">Explode to room-nights once, in the pipeline</h3>



<p class="wp-block-paragraph">The fact table that makes everything else easy has one row per room, per night, per reservation. Stay dates, rate code, market segment, source of business, channel, room type, rate amount and the taxes as separate columns. Every downstream metric becomes a filter and a sum instead of a range join.</p>



<p class="wp-block-paragraph">Do the explosion in the transform layer, not in the BI tool. If a range join lives in a Power BI model or a QuickSight dataset, it exists once per report and drifts once per report.</p>



<h3 class="wp-block-heading">The denominator is a decision, not a fact</h3>



<p class="wp-block-paragraph">&#8220;Rooms available&#8221; sounds like a constant. It isn&#8217;t. Rooms out of order for renovation, rooms out of service for a night, house-use rooms and comp rooms all get treated differently by different properties, and the PMS report you&#8217;re reconciling against has already made a choice.</p>



<p class="wp-block-paragraph">Same question on the numerator. Whether no-show revenue counts as room revenue, whether cancellation fees count, whether package components get split out of the room rate, whether ADR is quoted net or gross of tax. Hotel accounting conventions follow the Uniform System of Accounts for the Lodging Industry, and most PMS vendors document how their own reports treat each case. Read that documentation before you write a single SQL definition.</p>



<p class="wp-block-paragraph">Write the definitions down in the repository next to the transform code. When the lake and the PMS disagree later, the first question is always whether it&#8217;s a bug or a definition, and without written definitions you cannot answer that question quickly.</p>



<h2 class="wp-block-heading">Model restatements with two dates, not one</h2>



<p class="wp-block-paragraph">Every row needs a business date, which is the night the revenue belongs to, and an as-of timestamp, which is when you learned it. Two time axes. That&#8217;s what makes the difference between &#8220;the numbers changed&#8221; and &#8220;here is exactly what changed, when, and by how much&#8221;.</p>



<p class="wp-block-paragraph">There are two ways to store that. Keep every version of every row and let the reader pick a version, or keep the current truth in a table you update in place and keep the history separately. The second is easier to query and easier to explain, and it&#8217;s the one I reach for first. It needs a table format that supports updates and deletes, which in practice on AWS means Apache Iceberg.</p>



<p class="wp-block-paragraph">Iceberg gives you three things that matter here: row-level MERGE, snapshots you can query as of a point in time, and schema evolution without rewriting the dataset. Athena, Glue ETL, EMR and Redshift all read Iceberg tables through the Glue Data Catalog, so you&#8217;re not tying the storage to one engine.</p>



<p class="wp-block-paragraph">A curated room-night table in Athena looks roughly like this. Note the partitioning, which I&#8217;ll come back to.</p>



<pre class="wp-block-code"><code>CREATE TABLE curated.room_night (
  property_code      string,
  business_date      date,
  confirmation_id    string,
  room_number        string,
  room_type_code     string,
  rate_code          string,
  market_segment     string,
  source_of_business string,
  room_revenue       decimal(12,2),
  room_tax           decimal(12,2),
  reservation_status string,
  guest_key          string,
  as_of_ts           timestamp,
  source_system      string
)
PARTITIONED BY (property_code, month(business_date))
LOCATION 's3://example-lake-curated/room_night/'
TBLPROPERTIES ('table_type' = 'ICEBERG');</code></pre>



<p class="wp-block-paragraph">Loading is a MERGE from a staging table holding the latest extract. Matched rows get overwritten with the newer version, unmatched rows get inserted, and rows that vanished from the source get marked rather than deleted.</p>



<pre class="wp-block-code"><code>MERGE INTO curated.room_night AS t
USING staging.room_night_batch AS s
  ON  t.property_code   = s.property_code
  AND t.business_date   = s.business_date
  AND t.confirmation_id = s.confirmation_id
  AND t.room_number     = s.room_number
WHEN MATCHED AND s.as_of_ts &gt; t.as_of_ts THEN
  UPDATE SET
    room_revenue       = s.room_revenue,
    room_tax           = s.room_tax,
    rate_code          = s.rate_code,
    reservation_status = s.reservation_status,
    as_of_ts           = s.as_of_ts
WHEN NOT MATCHED THEN
  INSERT (property_code, business_date, confirmation_id, room_number,
          room_revenue, room_tax, rate_code, reservation_status, as_of_ts)
  VALUES (s.property_code, s.business_date, s.confirmation_id, s.room_number,
          s.room_revenue, s.room_tax, s.rate_code, s.reservation_status, s.as_of_ts);</code></pre>



<p class="wp-block-paragraph">The <code>as_of_ts</code> comparison in the MATCHED clause is doing real work. Extracts arrive out of order more often than you&#8217;d like, particularly when a backfill runs alongside the scheduled job, and without that guard a late-arriving older snapshot overwrites newer truth. It&#8217;s a silent corruption, which is the worst kind.</p>



<h3 class="wp-block-heading">Keep a restatement log, not just the current state</h3>



<p class="wp-block-paragraph">Before the MERGE runs, write the delta out to an append-only table: key, field changed, old value, new value, as-of timestamp. It costs almost nothing and it turns &#8220;the March numbers moved&#8221; into a query that returns rows.</p>



<p class="wp-block-paragraph">You can also lean on Iceberg snapshots and time travel for this, and they&#8217;re useful for spot checks. I still prefer an explicit log, because snapshot retention will eventually expire the snapshot you want, and finance will eventually ask about a night that&#8217;s older than your retention window.</p>



<h2 class="wp-block-heading">Landing the sources without building a point-to-point mess</h2>



<p class="wp-block-paragraph">A property runs more systems than people expect. PMS, POS, channel manager, revenue management, spa and golf, parking, loyalty, and whatever the marketing team bought last year. Land each of them raw and immutable first, then transform. The raw zone is your audit trail and your ability to reprocess without going back to a vendor API.</p>



<h3 class="wp-block-heading">The PMS is the hard one</h3>



<p class="wp-block-paragraph">For OPERA Cloud, the Oracle Hospitality Integration Platform is the supported route. It exposes REST APIs over OAuth, and registration is self-service rather than going through the older partner validation process. Two things to plan for: the API surface is large and reservation, financial and configuration data live in different areas, and access depends on the property&#8217;s licensing, so confirm what the property actually has before you scope the work.</p>



<p class="wp-block-paragraph">Legacy on-premise OPERA is a different world. Integration there goes through the older interface stack and the validation programme, and scheduled report exports to a file drop are frequently the pragmatic answer. Cloud-native systems like Cloudbeds and Mews expose REST APIs directly and are far less work.</p>



<p class="wp-block-paragraph">Reading the PMS database directly, where you can even reach it, is tempting and usually a bad idea. Schema changes arrive with vendor upgrades and nothing tells you in advance, and depending on the contract it can put support out of scope. Check the agreement before you build anything on top of direct database access.</p>



<h3 class="wp-block-heading">The relay box, if you need one</h3>



<p class="wp-block-paragraph">Properties with on-premise systems usually need something that picks up scheduled exports and pushes them to S3. Keep it boring: a small always-on Linux host, a scheduled job, checksum on write, and an SNS or CloudWatch alarm when a file doesn&#8217;t turn up. A cheap VPS from somewhere like Contabo or InterServer does this fine, and it is much easier to hand over at the end of an engagement than a machine sitting under the front desk.</p>



<p class="wp-block-paragraph">For reaching that host without opening inbound ports on a hotel network, an outbound-only tunnel such as Cloudflare Tunnel is a clean fit. If staff are hitting admin interfaces from guest or public Wi-Fi, a consumer VPN client like NordVPN or Surfshark protects that transport, though it is not a substitute for a real network path into the property.</p>



<h3 class="wp-block-heading">Time zones will bite you</h3>



<p class="wp-block-paragraph">The hotel business date is not a calendar day and it is not UTC. It&#8217;s a property-local concept that rolls at night audit, which might be at 2am or 4am depending on the property, and it can be held open when the audit runs late.</p>



<p class="wp-block-paragraph">Store the source system&#8217;s own business date as the partition key. Store event timestamps in UTC alongside it. Never derive the business date by truncating a UTC timestamp, because for a group with properties in several time zones that quietly shifts revenue between nights, and the error is small enough to look like noise.</p>



<h2 class="wp-block-heading">Keep card data out and plan for erasure on day one</h2>



<p class="wp-block-paragraph">Guest records are personal data and hotels hold a lot of it: names, addresses, passport and ID details in some jurisdictions, loyalty identifiers, stay history. Payment card data sits under a separate regime again.</p>



<p class="wp-block-paragraph">The single highest-value decision is negative. Do not bring cardholder data into the analytics platform at all. Nothing in occupancy, ADR or segment analysis needs a card number, and pulling one in drags the whole lake, its query engines and everyone with access into PCI DSS scope. Take the payment token or a masked reference from the PMS and stop there.</p>



<p class="wp-block-paragraph">For guest identity, pseudonymise at ingestion. Keep a surrogate guest key in the analytics tables and hold the mapping to real identity in one small, separately governed place with its own access controls. Amazon Macie can scan the raw zone for personal data that shouldn&#8217;t be there, which is useful precisely because someone will eventually land a CSV export nobody reviewed.</p>



<p class="wp-block-paragraph">Erasure needs thinking about before you have to do it. Iceberg makes row-level deletes straightforward, but a deleted row is still present in older snapshots, and time travel will happily return it. Deletion isn&#8217;t complete until snapshot expiration has run past the snapshots containing that row and the underlying files have been removed. Set retention deliberately, know what your window is, and be able to state it. Crypto-shredding, where the identity mapping is encrypted with a per-subject key you can destroy, is the cleaner design if the requirement is strict.</p>



<p class="wp-block-paragraph">The same discipline applies to workstations. Analysts pull extracts, extracts contain guest names, and laptops get replaced. A secure erasure tool such as O&amp;O SafeErase covers that end of it, and it belongs in the same policy as the lake.</p>



<p class="wp-block-paragraph">For access control, Lake Formation is the right layer. Define column and row filters once and they&#8217;re enforced across Athena, EMR and Redshift rather than reimplemented per engine. A group operator can then give each property&#8217;s management team row access to its own data from the same tables, which is far less work than building a schema per property.</p>



<h2 class="wp-block-heading">File physics at hotel scale</h2>



<p class="wp-block-paragraph">Hotel data is small. A single property produces a few hundred room-nights a day. Even a large group is nowhere near the volumes that data lake tutorials assume, and that changes the advice in one specific direction: your problem is too many tiny files, not too much data.</p>



<p class="wp-block-paragraph">Two habits create it. Partitioning by day when a property generates kilobytes per day, and polling the PMS every fifteen minutes so each run writes another sliver. Both feel careful and both make queries slower and more expensive, because the engine spends its time opening files rather than reading them.</p>



<ul class="wp-block-list">
<li>Partition curated tables by property and month, not by day. Iceberg prunes from its own metadata, so you get file-level skipping without the partition explosion.</li>

<li>Batch the writes. Buffer frequent polls and MERGE on a schedule that matches how the numbers are actually consumed, which for most operational reporting means hourly at most.</li>

<li>Run compaction. The Glue Data Catalog offers managed table optimizers for Iceberg tables covering compaction, snapshot retention and orphan file deletion, configurable from the console, CLI or API. Check the current format and engine support against the AWS docs before you rely on it.</li>

<li>Or drive it from Athena directly with <code>OPTIMIZE</code> to rewrite data files and <code>VACUUM</code> to expire snapshots and remove orphans.</li>
</ul>



<p class="wp-block-paragraph">For the raw zone, where tables are usually plain Hive-style Parquet rather than Iceberg, Athena partition projection is worth knowing about. It computes partition values from a configured pattern instead of listing them from the catalog, which removes a metadata round trip on tables with many partitions. It does not apply to Iceberg tables, which handle pruning through their own manifests.</p>



<p class="wp-block-paragraph">Athena bills on data scanned and Glue on job runtime, so cost tracks file layout and query shape rather than how much data you own. That&#8217;s a good thing at this scale, but it does mean a single badly written dashboard query on a schedule can cost more than the entire storage bill. Tools like Vantage or CloudZero make that visible per workload if the AWS cost console isn&#8217;t giving you enough granularity.</p>



<h2 class="wp-block-heading">Troubleshooting a hotel data lake on AWS</h2>



<p class="wp-block-paragraph"><strong>The lake and the PMS report disagree by a small, consistent amount.</strong> Almost always a definition, not a bug. Compare the room count first: if rooms sold matches and revenue doesn&#8217;t, you&#8217;re including or excluding tax, packages or fees differently. If rooms sold doesn&#8217;t match, check house-use and comp rooms.</p>



<p class="wp-block-paragraph"><strong>Yesterday reconciles, last month doesn&#8217;t.</strong> The append-only failure. Query your restatement log for that period. If you don&#8217;t have one, compare the current curated table against an older Iceberg snapshot for the same business dates.</p>



<p class="wp-block-paragraph"><strong>Numbers went backwards after a backfill.</strong> An older extract overwrote a newer one. Check that the as-of guard is present in the MERGE and that the staging table is deduplicated to one row per key before the merge runs, keeping the highest as-of value.</p>



<p class="wp-block-paragraph"><strong>Occupancy exceeds one hundred percent.</strong> Usually duplicate room-nights from a reservation that was modified into a new record while the old one stayed active, or a room moved mid-stay and counted in both rooms for the same night. Add a uniqueness check on property, business date and room number to the pipeline and fail the load rather than publishing it.</p>



<p class="wp-block-paragraph"><strong>Queries got slow and nothing changed.</strong> Look at file counts before anything else. Compaction that stopped running, or a new high-frequency job appending small files, explains most of these.</p>



<p class="wp-block-paragraph"><strong>A daily load silently produced nothing.</strong> This is the one that hurts, because an empty successful run looks like a healthy pipeline. Alert on the absence of expected rows per property per business date, not just on job failure. A deadman check in CloudWatch, or in Grafana Cloud if you already run dashboards there, catches it the same morning instead of at month end.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Storing reservations instead of room-nights, then rebuilding the date logic in every report.</li>

<li>Treating the ingestion date as the business date. They diverge the first time a load fails and gets rerun the next morning.</li>

<li>Building the reporting layer before agreeing the metric definitions with the finance and revenue teams.</li>

<li>Pulling the full guest profile because the API returns it, rather than the fields the analysis needs.</li>

<li>Skipping the raw zone and transforming on ingest, which means a logic bug is unrecoverable without re-extracting from the vendor.</li>

<li>Partitioning by day out of habit, then wondering why a small dataset queries slowly.</li>

<li>Relying on time travel as the audit trail, and discovering retention expired the snapshot finance asked about.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One row per room, per night, per reservation. Everything else derives from it.</li>

<li>Business date and as-of timestamp on every fact row, always.</li>

<li>Raw zone immutable, curated zone merged, consumption zone shaped for the tool that reads it.</li>

<li>Metric definitions in version control, next to the SQL that implements them.</li>

<li>A daily reconciliation query against one PMS report, checked automatically, with a tolerance and an alert.</li>

<li>Pseudonymise guest identity at ingestion and keep the mapping somewhere separate.</li>

<li>Compaction and snapshot retention configured before the table grows, not after someone complains.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Iceberg, or is plain Parquet enough?</h3>



<p class="wp-block-paragraph">Plain Parquet is fine for anything genuinely append-only, such as raw extracts and event logs. It stops being fine the moment you need to correct history, because updating a row means rewriting whole partitions and coordinating that yourself. Since correcting history is the defining characteristic of hotel data, the curated layer wants a table format with row-level updates.</p>



<h3 class="wp-block-heading">Should the data lake match the PMS exactly?</h3>



<p class="wp-block-paragraph">It should match one named PMS report, on one named metric, within a tolerance you&#8217;ve agreed. Trying to match every report is a trap, because vendor reports use different denominators and different treatment of taxes and fees, and several of them disagree with each other. Pick the report finance already trusts and reconcile to that one.</p>



<h3 class="wp-block-heading">How often should the pipeline pull from the PMS?</h3>



<p class="wp-block-paragraph">Match the decision it feeds. Financial reporting is a daily cycle anchored on night audit and needs nothing faster. Operational views like arrivals and room status benefit from intraday refresh, but that&#8217;s a different table with a different retention policy, not a reason to run the whole lake hourly.</p>



<h3 class="wp-block-heading">Can guest names go in the data lake?</h3>



<p class="wp-block-paragraph">Sometimes, with a lawful basis, retention limits and access controls. But most of the analysis people actually want, occupancy, rate, segment, channel mix, pace, works perfectly on a pseudonymous key. Start without names and add them back only for a use case that genuinely requires them. Card data is a different matter and the answer there is simply no.</p>



<h3 class="wp-block-heading">Athena or Redshift for the reporting layer?</h3>



<p class="wp-block-paragraph">At single-property and small-group volumes, Athena over Iceberg is usually enough, and there&#8217;s no cluster to size or keep warm. Redshift earns its place when you have high-concurrency BI users, heavy joins across many domains, or an existing warehouse investment. Either way, keep the storage open so the choice stays reversible.</p>



<h3 class="wp-block-heading">How do I handle a group with many properties?</h3>



<p class="wp-block-paragraph">One set of tables with property code as the leading partition key, and Lake Formation row filters for per-property access. Separate schemas per property look tidy at three properties and become unmanageable at thirty, because every schema change and every metric fix has to be applied thirty times.</p>



<h3 class="wp-block-heading">Where does benchmarking data fit?</h3>



<p class="wp-block-paragraph">Land it in the same lake as a separate domain, at whatever grain the provider supplies, usually a market aggregate rather than a room-night. Keep it in its own tables and join at report time. Mixing external benchmark figures into your own fact table is how index numbers end up in a revenue total.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A hotel data lake on AWS is not hard because of the volume. It&#8217;s hard because the past keeps changing, and an architecture that assumes otherwise fails quietly and slowly rather than loudly and fast.</p>



<p class="wp-block-paragraph">Get the room-night grain right, carry a business date and an as-of timestamp on every row, merge instead of appending, and write down what each metric means before anyone builds a dashboard on it. The S3, Glue, Iceberg and Athena assembly is the easy part, and it&#8217;s well documented. Reconciling with the night audit six months later is the part that decides whether anyone trusts the thing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of the hospitality data work I get asked about is one of a small number of shapes. Here&#8217;s where I&#8217;d be useful:</p>



<ul class="wp-block-list">
<li>Designing the room-night model and the bitemporal load, including the MERGE logic and the restatement log, so your numbers reconcile with the night audit and you can show why they moved.</li>

<li>Getting data out of the PMS reliably, whether that&#8217;s OHIP against OPERA Cloud, a REST API on a cloud-native system, or a scheduled export relay for an on-premise property.</li>

<li>Auditing an existing lake that has quietly drifted, finding which business dates restated and which metric definitions diverged from the reports finance uses.</li>

<li>Cutting Athena and Glue spend on a small dataset that behaves like a big one, usually through partition layout, compaction and killing scheduled queries nobody reads.</li>

<li>Setting up Lake Formation row and column controls for a multi-property group so each property sees its own data from shared tables.</li>

<li>Building the boring safety net: freshness and row-count checks per property per business date, deadman alerts, and a daily automated reconciliation against a named PMS report.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a discrepancy you can&#8217;t explain, send me the two figures and the query behind one of them, and I&#8217;ll tell you which side is wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/">Building a Hotel Data Lake on AWS That Agrees With the Night Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</title>
		<link>https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 20 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Apache Spark]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Snapshot Expiration]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=855</guid>

					<description><![CDATA[<p>Your lake bucket keeps growing even though nobody added data. Here is where the bytes actually sit: snapshots you never expired, orphan files no metadata points at, metadata JSON accumulation, and the object store's own hidden inventory. Plus the safe order to clean all four up.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/">Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually comes from finance, not engineering. Someone points at the bucket holding the analytics lake and asks why it is twice the size it was two quarters ago. Nobody added a pipeline. The tables are the same tables. One of them had a large chunk of history deleted on purpose, and the bucket still grew.</p>



<p class="wp-block-paragraph">That gap between what a table logically contains and what the object store physically holds is what Iceberg table maintenance exists to close. Iceberg never overwrites a file in place. Deletes and updates write new files and a new pointer, and the old files sit where they are until something explicitly removes them. If nothing does, the bill only moves one way.</p>



<p class="wp-block-paragraph">This post covers where those bytes hide, why expiring snapshots often frees nothing, how orphan files accumulate without showing up in any metadata you would think to check, and the order to run cleanup so you do not corrupt a table while shrinking it.</p>



<h2 class="wp-block-heading">Why deleting rows does not shrink anything</h2>



<p class="wp-block-paragraph">Every write produces a snapshot: an immutable view of the table at a point in time, pointing at a manifest list, which points at manifests, which point at data files. Nothing in that chain is mutated. A new commit builds a new chain that reuses most of the old files.</p>



<p class="wp-block-paragraph">That design is what gives you time travel, rollback and safe concurrent writes. It is also why storage grows silently. A data file only becomes a deletion candidate once <em>no</em> retained snapshot references it. Delete a year of rows and you free nothing, because the snapshot from before the delete still points at every one of those files.</p>



<p class="wp-block-paragraph">So maintenance is not an optimization. It is the mechanism by which deletes become real.</p>



<h2 class="wp-block-heading">The four places your storage is actually sitting</h2>



<p class="wp-block-paragraph">When a lakehouse bucket balloons, the bytes are in one of four categories. Each needs a different tool, and confusing them is how teams end up running the same job for weeks with no effect.</p>



<h3 class="wp-block-heading">1. Data files held alive by snapshots you never expired</h3>



<p class="wp-block-paragraph">This is the one everyone knows about and usually the largest share. Snapshots accumulate until something expires them, and nothing expires them by default.</p>



<p class="wp-block-paragraph">Before changing anything, look at the table&#8217;s own metadata. Iceberg exposes it as queryable tables:</p>



<pre class="wp-block-code"><code>-- How many snapshots exist, and how far back do they go?
SELECT committed_at, snapshot_id, operation
FROM db.orders.snapshots
ORDER BY committed_at;

-- How much data does the CURRENT snapshot reference?
SELECT count(*) AS file_count,
       sum(file_size_in_bytes) AS bytes
FROM db.orders.files;</code></pre>



<p class="wp-block-paragraph">Compare that second number against what your object store reports for the table prefix. The difference is your maintenance debt. On a neglected table the referenced set is often a minority of the physical bytes.</p>



<p class="wp-block-paragraph">Expiring snapshots is a Spark stored procedure:</p>



<pre class="wp-block-code"><code>CALL my_catalog.system.expire_snapshots(
  table =&gt; 'db.orders',
  older_than =&gt; TIMESTAMP '2000-01-01 00:00:00',
  retain_last =&gt; 10,
  stream_results =&gt; true
);</code></pre>



<p class="wp-block-paragraph">Three arguments matter. <code>older_than</code> sets the age cutoff. <code>retain_last</code> is the floor: it keeps that many ancestor snapshots regardless of age, which is your rollback safety net. <code>stream_results</code> streams the delete list back to the driver instead of collecting it at once, which on a table with long history is the difference between the job finishing and the driver running out of memory.</p>



<p class="wp-block-paragraph">The trap is that Iceberg treats these conditions as guidance rather than a strict contract, leaning toward retention when they conflict. An aggressive <code>older_than</code> plus a generous <code>retain_last</code> gives you the generous answer. Safe, but it surprises people who expected the timestamp to win.</p>



<p class="wp-block-paragraph">Branches and tags are separate: they are not subject to snapshot age at all. A tag created for a regulatory export a year ago pins its entire snapshot chain, and no amount of expiring will touch it. Reference age is governed by <code>history.expire.max-ref-age-ms</code>.</p>



<h3 class="wp-block-heading">2. Orphan files that no metadata points at</h3>



<p class="wp-block-paragraph">This is the invisible one, and the reason many teams conclude that maintenance &#8220;does not work.&#8221;</p>



<p class="wp-block-paragraph">When a Spark task writes a data file and the job then fails before the commit lands, that file is already on disk. It is in no manifest, no snapshot, no metadata file. Expiry will never find it, because expiry compares the file sets of expired snapshots against retained ones, and a file that was never in a snapshot is in neither set.</p>



<p class="wp-block-paragraph">Streaming writers, speculative execution, retried stages and killed compaction jobs all produce these. The only way to find them is to list the storage prefix and diff it against the metadata:</p>



<pre class="wp-block-code"><code>-- Always dry run first. This lists candidates without deleting.
CALL my_catalog.system.remove_orphan_files(
  table =&gt; 'db.orders',
  dry_run =&gt; true
);</code></pre>



<p class="wp-block-paragraph">Read that output before you run it live. Recent timestamps or unfamiliar paths mean stop and investigate.</p>



<p class="wp-block-paragraph">The safety mechanism is <code>older_than</code>, which defaults to three days. That default is not arbitrary caution. Orphan removal cannot distinguish a file abandoned by a dead job from one being written right now by a live one. Set the interval shorter than your longest-running write and the procedure can delete a file a job is about to commit, corrupting the table. Shorten it only when you know your write durations, and never below your longest backfill.</p>



<p class="wp-block-paragraph">The second hazard is less obvious. Iceberg compares file paths as strings. If your metadata records <code>s3a://bucket/...</code> and your listing returns <code>s3://bucket/...</code>, or you migrated between HDFS namenodes and the authority changed, every live file looks like an orphan. Running that deletes the table. The procedure defends against it by erroring on a prefix mismatch by default, via <code>prefix_mismatch_mode</code> with <code>ERROR</code>, <code>IGNORE</code> and <code>DELETE</code> as options. Resolve a mismatch with <code>equal_schemes</code> or <code>equal_authorities</code>. Do not reach for <code>DELETE</code> to silence it.</p>



<h3 class="wp-block-heading">3. Metadata JSON files and manifests</h3>



<p class="wp-block-paragraph">Every commit writes a new table metadata JSON file. On a batch table that is a handful a day. On a streaming table committing every minute it is a lot of small objects, and those cost you in request charges and planning time as much as in bytes.</p>



<pre class="wp-block-code"><code>ALTER TABLE db.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '20'
);</code></pre>



<p class="wp-block-paragraph">The first property is off by default. The second defaults to 100 and controls how many previous metadata files are <em>tracked</em> in the metadata log. The catch is that cleanup only deletes tracked files. If you have been running with deletion disabled, files that fell out of the log are already untracked, and turning the property on later will not reach back for them. They are orphans now, and only orphan file removal clears them.</p>



<p class="wp-block-paragraph">You can see the tracked set with <code>SELECT timestamp, file FROM db.orders.metadata_log_entries</code>.</p>



<p class="wp-block-paragraph">Manifests rarely dominate the bill, but a table with thousands of tiny ones plans queries slowly. <code>rewrite_manifests</code> consolidates them. Treat it as a query performance job and run it after the deletion work, so it is not reorganizing metadata about to be thrown away.</p>



<h3 class="wp-block-heading">4. The object store&#8217;s own hidden inventory</h3>



<p class="wp-block-paragraph">Most Iceberg guides skip this category, and it often explains a bill nobody can account for. Iceberg has no visibility into any of it.</p>



<ul class="wp-block-list">
<li><strong>Incomplete multipart uploads.</strong> Large Parquet files upload in parts. When a writer dies partway, the completed parts sit in the bucket, billed as storage, and they do not appear in a normal object listing. Nothing removes them automatically.</li>



<li><strong>Noncurrent object versions.</strong> With versioning enabled, every file your maintenance job &#8220;deletes&#8221; becomes a noncurrent version you keep paying for, indefinitely, unless a rule expires it.</li>



<li><strong>Delete markers.</strong> Deletes on a versioned bucket leave markers behind. Small individually, but on a table with millions of expired files they add up and slow listings down.</li>
</ul>



<p class="wp-block-paragraph">A lifecycle configuration covers all three. On S3 the shape looks like this:</p>



<pre class="wp-block-code"><code>{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 14 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Now the warning that matters most here. <strong>Never put an age-based expiration or storage class transition rule on the prefix holding live Iceberg data files.</strong> A rule deleting objects older than ninety days will happily delete data files a current snapshot still references, and the table breaks with missing-file errors on the next read. Transitioning them to an archive class is just as bad, because reads then fail or stall on retrieval.</p>



<p class="wp-block-paragraph">The rules above act only on things the table does not reference: abandoned upload parts, superseded versions, tombstones. Iceberg decides what live data to delete; lifecycle rules clean up after that decision.</p>



<p class="wp-block-paragraph">This applies equally on S3-compatible storage. If you run a lakehouse on Cloudflare R2, Backblaze B2, Wasabi or Contabo Object Storage to dodge egress charges, check which lifecycle actions the provider actually implements. Support for aborting incomplete multipart uploads varies, and a provider that silently ignores the rule leaves you paying for parts you cannot see.</p>



<h2 class="wp-block-heading">Running Iceberg table maintenance in a safe order</h2>



<p class="wp-block-paragraph">Order matters, and the reason is not stylistic. These procedures read each other&#8217;s inputs.</p>



<ol class="wp-block-list">
<li><strong>Compact first, if you are compacting.</strong> <code>rewrite_data_files</code> writes new files and leaves the old ones referenced by older snapshots. Doing it after expiry means the space you just freed is immediately replaced.</li>



<li><strong>Expire snapshots.</strong> This is what turns unreferenced data files into deleted ones. It must happen before orphan removal, because expiry needs to read the manifests belonging to the snapshots it is dropping.</li>



<li><strong>Remove orphan files.</strong> Only after expiry has committed. Running the two concurrently is a genuine race: orphan removal can delete files the in-flight expiry job still needs to read, and expiry fails.</li>



<li><strong>Rewrite manifests, if planning is slow.</strong> Optional, and worth measuring before and after rather than running on faith.</li>
</ol>



<p class="wp-block-paragraph">On cadence: expiry is cheap and should run often, daily or better on high-commit tables. Orphan removal is expensive because it lists the entire table prefix, and the official guidance is to run it periodically but not frequently. Weekly on a busy table, monthly on a quiet one. Where supported, the <code>prefix_listing</code> option makes that scan substantially cheaper on object stores, where recursive directory listing is the slow part.</p>



<h2 class="wp-block-heading">Letting table properties do the boring part</h2>



<p class="wp-block-paragraph">Procedures are what you run. Properties are what you set once so the procedures behave sensibly without a config file per table.</p>



<pre class="wp-block-code"><code>ALTER TABLE db.orders SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms' = '604800000',   -- 7 days
  'history.expire.min-snapshots-to-keep' = '10'
);</code></pre>



<p class="wp-block-paragraph">The snapshot age default is five days and the minimum to keep defaults to one. Setting both means a generic maintenance job can call <code>expire_snapshots</code> with no tuning and each table gets its own policy. A regulated reporting table keeps ninety days; a staging table keeps an hour. That beats maintaining per-table retention in your scheduler.</p>



<p class="wp-block-paragraph">The other property worth knowing is <code>gc.enabled</code>. Setting it false blocks physical deletion entirely. It exists for tables whose files are shared with something else, and it is correct for those. It is also a common accidental cause of &#8220;maintenance runs successfully and frees nothing.&#8221; Check it before debugging anything else.</p>



<h2 class="wp-block-heading">Troubleshooting: the job succeeded and nothing shrank</h2>



<ul class="wp-block-list">
<li><strong>The counters were zero.</strong> <code>expire_snapshots</code> returns counts of deleted data files, delete files, manifests and manifest lists. All zero means nothing qualified. Check whether <code>retain_last</code> or the table&#8217;s minimum-to-keep property is holding everything.</li>



<li><strong>A branch or tag is pinning history.</strong> Refs do not age out with snapshots. Look at the table&#8217;s refs metadata table and see what is anchored.</li>



<li><strong>Garbage collection is off.</strong> Either <code>gc.enabled</code> is false, or the table was created by the <code>snapshot</code> procedure, which is barred from expiry precisely because it does not own its data files.</li>



<li><strong>Files were deleted but the bucket did not shrink.</strong> Versioning. The deletes created noncurrent versions. This is the most common cause of healthy counters and a flat bill.</li>



<li><strong>Orphan removal hit a file limit.</strong> Some managed implementations cap deletions per run. On a backlogged bucket you may need several passes before the number stops dropping.</li>



<li><strong>The driver died.</strong> On a table with very long history, add <code>stream_results</code> and raise <code>max_concurrent_deletes</code> so deletes spread across a thread pool instead of serializing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running orphan file removal with a short retention window on a table that has long-running writers. This is the one that can actually destroy data.</li>



<li>Setting a bucket-wide expiration rule to &#8220;clean up the lake&#8221; and taking live data files with it.</li>



<li>Enabling metadata deletion after the fact and expecting it to collect the already-untracked backlog.</li>



<li>Scheduling expiry and orphan removal as two independent cron jobs that can overlap.</li>



<li>Skipping the dry run because the last twenty were fine. The one that matters is the run after somebody changes the storage endpoint.</li>



<li>Setting retention from a storage target rather than a recovery target. Ask how far back you need to roll back, then keep that much.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth adopting</h2>



<ul class="wp-block-list">
<li>Express retention as a table property, not a job argument. It travels with the table and survives whoever wrote the scheduler.</li>



<li>Track referenced bytes against physical bytes per table as a metric, not an occasional query. The ratio drifting upward is the early signal; the invoice is the late one. Any time series backend you already run will do.</li>



<li>Give maintenance its own compute and its own credentials. It is the only workload in the lake that needs delete permissions.</li>



<li>Log the returned counters from every run. When someone asks in six months where the storage went, that log is the answer.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How often should I expire Iceberg snapshots?</h3>



<p class="wp-block-paragraph">As often as your commit rate justifies. A streaming table committing every minute benefits from daily or hourly expiry; a nightly batch table is fine with weekly. Set the window from how far back you would realistically need to roll back, then keep a few extra snapshots as a floor.</p>



<h3 class="wp-block-heading">Does expiring snapshots delete my data?</h3>



<p class="wp-block-paragraph">It deletes data files that no retained snapshot references. Current table contents are never affected. What you lose is the ability to time travel to or roll back to the expired snapshots, so agree the window with anyone who queries the table as of a date.</p>



<h3 class="wp-block-heading">Why do orphan files exist if Iceberg commits are atomic?</h3>



<p class="wp-block-paragraph">The commit is atomic; the writes preceding it are not part of that atomicity. Data files are written first, then the commit makes them visible. If the job dies between those steps, the files exist and the commit never happened. Atomicity guarantees readers never see a partial table, not that failed writes clean up after themselves.</p>



<h3 class="wp-block-heading">Can I use an S3 lifecycle rule instead of Iceberg maintenance?</h3>



<p class="wp-block-paragraph">No, and trying is how tables get broken. Lifecycle rules act on object age. Whether Iceberg still needs a file has nothing to do with age; a data file written two years ago can be referenced by the current snapshot. Lifecycle rules are right for upload parts, noncurrent versions and delete markers, and wrong for anything the table might still point at.</p>



<h3 class="wp-block-heading">Do I need Spark to run Iceberg table maintenance?</h3>



<p class="wp-block-paragraph">Spark has the most complete set of procedures, but it is not the only route. Flink has its own maintenance API, Trino exposes expiry and orphan removal as table procedures, and several managed catalogs run scheduled maintenance for you. The mechanics are identical regardless of engine, because the work is defined by the table format rather than the compute. Only the syntax and the exposed arguments change.</p>



<h3 class="wp-block-heading">Is orphan file removal safe to automate?</h3>



<p class="wp-block-paragraph">Yes, once you have verified that the retention window comfortably exceeds your longest write and that the paths in your metadata match those your storage listing returns. Automate it after watching a few dry runs, not before. Erroring on a prefix mismatch is a stop signal, not something to configure away.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Iceberg table maintenance is not housekeeping you get to postpone. It is the step where logical deletes become physical ones, and until it runs, storage only grows. The bill is the visible symptom; the same neglect also makes query planning slower and recovery windows fuzzier.</p>



<p class="wp-block-paragraph">Set retention as a table property so the policy lives with the data. Expire snapshots on a schedule matching your commit rate. Remove orphan files less often, always after expiry, always with a window longer than your slowest write. Then add the storage lifecycle rules that clean up what Iceberg cannot see. Any one alone leaves money on the table.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your lakehouse storage under control?</h2>



<p class="wp-block-paragraph">Most of the work here is not writing the procedures. It is working out which of the four categories your bytes are in, and building a maintenance path that will not break a table at three in the morning. That is the kind of thing I do:</p>



<ul class="wp-block-list">
<li>Audit an existing lake and produce a per-table breakdown of referenced bytes against physical bytes, so you know where the money is before touching anything</li>



<li>Build a scheduled maintenance pipeline with correct sequencing, dry-run gates and per-table retention driven by table properties</li>



<li>Set retention against real rollback and time travel requirements, gathered from the people who depend on them</li>



<li>Design storage lifecycle rules that clear incomplete uploads, noncurrent versions and delete markers without touching live data files</li>



<li>Diagnose maintenance jobs that report success and free nothing, including prefix mismatches, pinned refs and disabled garbage collection</li>



<li>Add table health metrics and alerting so drift shows up as a graph rather than a surprise on the invoice</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of your snapshots metadata table, your current table properties, or the counters from a run that did not do what you expected. That is usually enough to tell you where the bytes are.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/">Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: Where the Money Actually Goes</title>
		<link>https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 31 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Predicate Pushdown]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=342</guid>

					<description><![CDATA[<p>A practical comparison of Parquet, JSON and CSV on cost and query performance, covering how each billing model changes the answer, the mechanics behind columnar savings, the failure modes that cancel them out, and a six-step way to model the migration before you commit to it.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A team converts its event logs from JSON to Parquet over a weekend. The Glue job runs clean, the files land in S3, the dashboards keep working. Monday morning someone opens the Athena bill expecting it to have fallen off a cliff, and it has moved by about a tenth.</p>



<p class="wp-block-paragraph">That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.</p>



<p class="wp-block-paragraph">This is a working comparison of <strong>Parquet vs JSON vs CSV</strong> on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.</p>



<h2 class="wp-block-heading">Your billing model decides how much the format matters</h2>



<p class="wp-block-paragraph">Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.</p>



<ul class="wp-block-list"><li><strong>Bytes scanned from object storage.</strong> Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.</li><li><strong>Logical bytes processed.</strong> BigQuery&#8217;s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.</li><li><strong>Compute time.</strong> Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.</li></ul>



<p class="wp-block-paragraph">The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query <em>external</em> data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.</p>



<h2 class="wp-block-heading">CSV: cheap to produce, expensive to interrogate</h2>



<p class="wp-block-paragraph">CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.</p>



<h3 class="wp-block-heading">Where CSV wins</h3>



<ul class="wp-block-list"><li>A human needs to open it. Nothing else comes close.</li><li>Export from a legacy system that offers no other option.</li><li>Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.</li><li>Streaming appends. You can write a line at a time and stop whenever you like.</li></ul>



<h3 class="wp-block-heading">Where CSV does not</h3>



<p class="wp-block-paragraph">Two problems, and the second is worse than the first.</p>



<p class="wp-block-paragraph">The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the <code>lineitem</code> table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.</p>



<p class="wp-block-paragraph">The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.</p>



<p class="wp-block-paragraph">The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether <code>NA</code> means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.</p>



<h2 class="wp-block-heading">JSON: you pay for the schema on every single row</h2>



<p class="wp-block-paragraph">Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called <code>customer_subscription_status</code> carries those thirty-odd bytes on every row, forever.</p>



<h3 class="wp-block-heading">Where JSON wins</h3>



<ul class="wp-block-list"><li>Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.</li><li>Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.</li><li>Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.</li><li>API and webhook payloads, where it is simply what arrives.</li></ul>



<h3 class="wp-block-heading">Where JSON does not</h3>



<p class="wp-block-paragraph">As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.</p>



<p class="wp-block-paragraph">The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.</p>



<h2 class="wp-block-heading">Parquet: what the file is actually doing</h2>



<p class="wp-block-paragraph">Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.</p>



<p class="wp-block-paragraph">A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:</p>



<ul class="wp-block-list"><li><strong>Column pruning.</strong> The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.</li><li><strong>Predicate pushdown.</strong> If a row group&#8217;s maximum value for <code>event_date</code> falls below your filter&#8217;s lower bound, the entire row group is skipped without being read.</li></ul>



<p class="wp-block-paragraph">Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group&#8217;s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes <code>parquet_bloom_probe</code> so you can check which row groups a given value would eliminate.</p>



<p class="wp-block-paragraph">Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.</p>



<h3 class="wp-block-heading">Where Parquet does not win</h3>



<ul class="wp-block-list"><li><strong>Point lookups.</strong> Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.</li><li><strong>Row-level updates.</strong> Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.</li><li><strong>Streaming writes.</strong> The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.</li><li><strong>Tiny datasets.</strong> Below a few megabytes, footer and row group overhead is not worth paying.</li><li><strong>Human inspection.</strong> It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the savings quietly disappear</h2>



<p class="wp-block-paragraph">Every one of these has cost someone a migration that looked excellent in the design document.</p>



<ul class="wp-block-list"><li><strong>Small files.</strong> Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.</li><li><strong>One enormous row group.</strong> Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.</li><li><strong>Select-everything queries in the reporting layer.</strong> Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.</li><li><strong>No partitioning.</strong> Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.</li><li><strong>Gzipped text.</strong> AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.</li><li><strong>Uppercase file extensions.</strong> A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as <code>.GZ</code>, and treats a file with no extension as uncompressed plain text.</li></ul>



<h2 class="wp-block-heading">How to model the change before you migrate</h2>



<p class="wp-block-paragraph">Do not migrate on faith. This takes an afternoon and answers the question properly.</p>



<ol class="wp-block-list"><li><strong>Find out what you scan today.</strong> Every Athena query execution carries a <code>DataScannedInBytes</code> statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.</li><li><strong>Convert one day of data, not all of it.</strong> A single partition is enough to measure with.</li><li><strong>Look inside the resulting file</strong> before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.</li><li><strong>Re-run the same queries</strong> against the converted partition and compare bytes scanned. Measured, not estimated.</li><li><strong>Price the write side.</strong> Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.</li><li><strong>Check who else reads the files.</strong> One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.</li></ol>



<p class="wp-block-paragraph">For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:</p>



<pre class="wp-block-code"><code>-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);</code></pre>



<p class="wp-block-paragraph">The <code>parquet_metadata</code> function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, <code>pyarrow.parquet.ParquetFile</code> exposes the same footer through its <code>metadata</code> attribute, including per-column statistics and whether a bloom filter offset is present.</p>



<p class="wp-block-paragraph">If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena&#8217;s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:</p>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;</code></pre>



<p class="wp-block-paragraph">Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.</p>



<h2 class="wp-block-heading">How I would decide</h2>



<ul class="wp-block-list"><li><strong>Keep the raw landing zone in whatever arrives.</strong> Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.</li><li><strong>Convert once, at the point where data becomes queryable.</strong> Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.</li><li><strong>Partition before you optimise the format.</strong> If there is budget for exactly one change, partition on the column your queries actually filter on.</li><li><strong>Choose the codec by read frequency.</strong> Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.</li><li><strong>Leave small CSVs alone.</strong> Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.</li><li><strong>Watch the scan bill, not the storage bill.</strong> Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Parquet always cheaper than CSV?</h3>



<p class="wp-block-paragraph">On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.</p>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV in practice?</h3>



<p class="wp-block-paragraph">It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.</p>



<h3 class="wp-block-heading">Does compressing my JSON or CSV give me the same savings?</h3>



<p class="wp-block-paragraph">Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.</p>



<h3 class="wp-block-heading">Should I use Parquet or ORC?</h3>



<p class="wp-block-paragraph">For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.</p>



<h3 class="wp-block-heading">Why did my costs not drop after converting to Parquet?</h3>



<p class="wp-block-paragraph">The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull <code>DataScannedInBytes</code> for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.</p>



<h3 class="wp-block-heading">Is JSON still worth keeping now that Parquet has a Variant type?</h3>



<p class="wp-block-paragraph">As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.</p>



<h3 class="wp-block-heading">What file size should I aim for?</h3>



<p class="wp-block-paragraph">Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.</p>



<p class="wp-block-paragraph">An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your data lake costs?</h2>



<p class="wp-block-paragraph">Most format work starts as a cost problem and ends as a layout problem. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all</li><li>Designing the partition scheme and file sizing before the conversion, so the migration is worth running</li><li>Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines</li><li>Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps</li><li>Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice</li><li>Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format</li></ul>



<p class="wp-block-paragraph">If you want a concrete opinion rather than a general one, send me a query&#8217;s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Backfilling Historical API Data into S3 Without Silent Gaps</title>
		<link>https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[S3 Inventory]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=268</guid>

					<description><![CDATA[<p>A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The backfill exits zero. Every window processed, nothing red in the logs, the object count in the bucket looks about right for the range. You close the ticket. Weeks later an analyst asks why the chart dips for one week in the middle of the historical range, and nobody can answer, because as far as every system involved is concerned, nothing failed.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn&#8217;t the one that throws. It&#8217;s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.</p>



<p class="wp-block-paragraph">This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.</p>



<h2 class="wp-block-heading">Why a backfill that finishes cleanly can still be wrong</h2>



<p class="wp-block-paragraph">A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it&#8217;s over. The only evidence it leaves behind is a set of objects in a bucket.</p>



<p class="wp-block-paragraph">So the question you need to answer months later isn&#8217;t &#8220;did the job succeed?&#8221; It&#8217;s &#8220;is this window complete, and how do I know?&#8221; If your design can&#8217;t answer that from artifacts on disk, you don&#8217;t have a backfill. You have a large amount of data of unknown provenance.</p>



<h2 class="wp-block-heading">Failure family 1: the source moved while you paginated</h2>



<p class="wp-block-paragraph">This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.</p>



<p class="wp-block-paragraph">If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<ul class="wp-block-list"><li><strong>Prefer keyset or cursor pagination.</strong> If the API offers an opaque cursor or a &#8220;records after ID X&#8221; parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can&#8217;t shift it.</li><li><strong>Sort by an immutable column.</strong> If you&#8217;re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.</li><li><strong>Bound every request on both ends.</strong> Don&#8217;t ask for &#8220;everything before now.&#8221; Ask for a closed interval with an upper bound already in the past. A closed window can&#8217;t grow while you read it.</li><li><strong>Know which timestamp the filter applies to.</strong> Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.</li></ul>



<p class="wp-block-paragraph">That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it&#8217;s worth writing down which one you chose.</p>



<h2 class="wp-block-heading">Failure family 2: the retry that wrote the same page twice</h2>



<p class="wp-block-paragraph">Backfills retry. They have to, because you&#8217;re hammering an API for hours and something will time out. The problem is what the retry writes.</p>



<p class="wp-block-paragraph">If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.</p>



<pre class="wp-block-code"><code>import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise</code></pre>



<p class="wp-block-paragraph">Two things in there are worth explaining rather than copying.</p>



<p class="wp-block-paragraph"><code>IfNoneMatch="*"</code> is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 <code>PreconditionFailed</code> otherwise. It&#8217;s supported on <code>PutObject</code> and <code>CompleteMultipartUpload</code>. That turns &#8220;did I already write this page?&#8221; into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 <code>ConditionalRequestConflict</code>, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.</p>



<p class="wp-block-paragraph">The same thing from the CLI, useful for testing the behaviour by hand:</p>



<pre class="wp-block-code"><code>aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"</code></pre>



<p class="wp-block-paragraph">The <code>retries</code> block matters for a reason that trips people up. In botocore, <code>max_attempts</code> means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a <code>Config</code> object it counts retries on top of the first. <code>total_max_attempts</code> exists only in a <code>Config</code> object and always means total, so using it removes the ambiguity. There&#8217;s also an <code>adaptive</code> mode that layers client-side rate limiting on top of <code>standard</code>; AWS still labels it experimental, so I reach for <code>standard</code> plus my own concurrency cap first.</p>



<h2 class="wp-block-heading">Failure family 3: a layout designed for writing, not reading</h2>



<p class="wp-block-paragraph">The prefix structure you pick during the backfill is the one you&#8217;re stuck with. Reorganising later means rewriting every object, and copy requests aren&#8217;t free.</p>



<h3 class="wp-block-heading">Partition on event time, not ingestion time</h3>



<p class="wp-block-paragraph">The tempting layout is the date you ran the job. It&#8217;s easy, it&#8217;s monotonic, and it&#8217;s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:</p>



<pre class="wp-block-code"><code>s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-&lt;token&gt;.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-&lt;n&gt;.parquet</code></pre>



<p class="wp-block-paragraph">Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.</p>



<h3 class="wp-block-heading">Don&#8217;t let the write path throttle itself</h3>



<p class="wp-block-paragraph">S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is <em>partitioned</em>. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 <code>SlowDown</code> responses even when your sustained rate looks well inside the documented ceiling.</p>



<p class="wp-block-paragraph">For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise <em>within</em> a single day and drive thousands of writes per second into one <code>dt=</code> prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.</p>



<h3 class="wp-block-heading">Register partitions without a scan</h3>



<p class="wp-block-paragraph">A backfill can create thousands of partitions at once. <code>MSCK REPAIR TABLE</code> handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.</p>



<p class="wp-block-paragraph">Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.</p>



<pre class="wp-block-code"><code>TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)</code></pre>



<p class="wp-block-paragraph">Replace <code>START_DATE</code> with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching <code>storage.location.template</code>, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.</p>



<h2 class="wp-block-heading">Failure family 4: the seam between backfill and live pipeline</h2>



<p class="wp-block-paragraph">Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.</p>



<p class="wp-block-paragraph">The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don&#8217;t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API&#8217;s own filtering will all conspire to open a hole at exactly T.</p>



<p class="wp-block-paragraph">Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.</p>



<h2 class="wp-block-heading">Failure family 5: the storage class you can&#8217;t undo</h2>



<p class="wp-block-paragraph">Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.</p>



<ul class="wp-block-list"><li>S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.</li><li>Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you&#8217;re charged the remainder.</li><li>Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.</li></ul>



<p class="wp-block-paragraph">Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you&#8217;re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.</p>



<p class="wp-block-paragraph">One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It&#8217;s a five-minute change people usually discover after the bill.</p>



<h2 class="wp-block-heading">Proving the backfill is complete</h2>



<p class="wp-block-paragraph">Here&#8217;s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.</p>



<pre class="wp-block-code"><code>{
  "dataset": "orders",
  "window": "&lt;dt value&gt;",
  "filter_field": "created_at",
  "range_start_inclusive": "&lt;iso8601&gt;",
  "range_end_exclusive": "&lt;iso8601&gt;",
  "pages_written": "&lt;int&gt;",
  "records_written": "&lt;int&gt;",
  "source_reported_total": "&lt;int or null&gt;",
  "last_cursor": "&lt;opaque token&gt;",
  "schema_fingerprint": "&lt;sha256 of sorted field names&gt;"
}</code></pre>



<p class="wp-block-paragraph">The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose <code>records_written</code> disagrees with <code>source_reported_total</code> is suspect. A window whose <code>schema_fingerprint</code> differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.</p>



<p class="wp-block-paragraph">Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.</p>



<h2 class="wp-block-heading">Troubleshooting a backfill that already ran</h2>



<ol class="wp-block-list"><li><strong>Count objects per window and look at the shape.</strong> Use S3 Inventory rather than a live <code>ListObjectsV2</code> walk on a large bucket. Plot objects per <code>dt=</code> partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.</li><li><strong>Reconcile record counts against the source.</strong> Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what&#8217;s in S3. If you kept manifests, this is one query.</li><li><strong>Check for duplicate natural keys within a partition.</strong> A <code>GROUP BY</code> on the record ID with <code>HAVING COUNT(*) &gt; 1</code> finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren&#8217;t deterministic.</li><li><strong>Look at boundary hours, not whole days.</strong> Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.</li><li><strong>Verify the layout still matches the table definition.</strong> If you&#8217;re using partition projection and a query returns zero rows, the first suspect is <code>storage.location.template</code> not matching the actual prefix, trailing slashes included.</li></ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating an exit code of zero as evidence of completeness.</li><li>Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.</li><li>Partitioning by ingestion date because it&#8217;s what the job naturally knows.</li><li>Making the backfill and live-ingest boundary exact instead of overlapping.</li><li>Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.</li><li>Running <code>MSCK REPAIR TABLE</code> over thousands of new partitions and trusting that it completed.</li><li>Backfilling on a last-modified field while assuming you captured creation events.</li><li>Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.</li></ul>



<h2 class="wp-block-heading">Best practices when you backfill historical API data into S3</h2>



<ul class="wp-block-list"><li>Make every window independently re-runnable. If re-running one day is routine rather than scary, you&#8217;ll actually fix problems when you find them.</li><li>Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.</li><li>Bound every API request on both ends, and never let the upper bound be &#8220;now&#8221;.</li><li>Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn&#8217;t possible.</li><li>Emit a manifest per window, written last, and reconcile it against source counts.</li><li>Compact to Parquet before applying any lifecycle transition.</li><li>Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won&#8217;t sleep halfway through.</li><li>Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I write raw JSON or convert to Parquet during the backfill?</h3>



<p class="wp-block-paragraph">Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don&#8217;t want to re-fetch years of data because of a type error.</p>



<h3 class="wp-block-heading">How large should each object be?</h3>



<p class="wp-block-paragraph">In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.</p>



<h3 class="wp-block-heading">Can I run the backfill in parallel across date windows?</h3>



<p class="wp-block-paragraph">Yes, and that&#8217;s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API&#8217;s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or <code>Retry-After</code> header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.</p>



<h3 class="wp-block-heading">Do I need AWS Glue or Airflow for this?</h3>



<p class="wp-block-paragraph">Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.</p>



<h3 class="wp-block-heading">What if the API caps how far back I can query?</h3>



<p class="wp-block-paragraph">Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it&#8217;s usually faster and less rate-limited than paginating.</p>



<h3 class="wp-block-heading">Does any of this change with S3-compatible storage instead of AWS?</h3>



<p class="wp-block-paragraph">The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with <code>If-None-Match</code>, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you&#8217;re depending on against that provider&#8217;s documentation rather than assuming parity with S3.</p>



<h3 class="wp-block-heading">How do I know the backfill is actually finished?</h3>



<p class="wp-block-paragraph">When every expected window has a manifest, every manifest&#8217;s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A backfill isn&#8217;t finished when the job exits. It&#8217;s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.</p>



<p class="wp-block-paragraph">If you&#8217;re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a backfill?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re staring at a historical range and an API with a rate limit, I can help with:</p>



<ul class="wp-block-list"><li>Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.</li><li>Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.</li><li>Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.</li><li>Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.</li><li>Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.</li><li>Wiring the seam between the backfill and your live pipeline so there&#8217;s no gap at the handover, plus the monitoring that would have caught it.</li></ul>



<p class="wp-block-paragraph">Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That&#8217;s usually enough to spot where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
