<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>S3 Tables | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/s3-tables/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/s3-tables/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 14 Nov 2026 07:01:31 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>S3 Tables | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/s3-tables/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 14 Nov 2026 07:01:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=683</guid>

					<description><![CDATA[<p>Managed Iceberg is not a storage upgrade, it is a decision about who owns compaction, snapshot expiry and orphan cleanup. A practical comparison of S3 Tables against Iceberg in general purpose S3 buckets, covering the maintenance you actually get, the Lake Formation trap in the access path, how the two bills are shaped, and a seven-step decision procedure you can run against a single table.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The dashboard that used to load in eight seconds now takes ninety. Nobody changed the query, nobody changed the schema, and the table is barely bigger than it was last quarter. Someone opens a ticket titled &#8220;Athena slow?&#8221; and it sits there, because there is no error to point at.</p>



<p class="wp-block-paragraph">What changed is the file count. A streaming job has been writing small Parquet files every few minutes for months, and each one is another entry the query planner opens before it can plan anything. The data is fine. The layout rotted.</p>



<p class="wp-block-paragraph">That is the problem underneath the whole S3 Tables vs Iceberg on S3 argument. It is not a storage question. It is a question about who owns table maintenance, and what happens when the answer is nobody. This post covers what S3 Tables actually runs for you, where each option wins, why the access path matters more than most comparisons admit, how the two bills are shaped, and a decision procedure you can apply to a specific table.</p>



<h2 class="wp-block-heading">The failure that doesn&#8217;t announce itself</h2>



<p class="wp-block-paragraph">Iceberg on general purpose S3 works beautifully on day one. You write Parquet, commit a snapshot, Athena reads it. Nothing about that first week tells you it needs upkeep. Three things degrade quietly afterward.</p>



<ul class="wp-block-list">
<li><strong>Small files accumulate.</strong> Planning cost scales with how many files the planner considers, not with how much data you actually scan.</li>



<li><strong>Snapshots pile up.</strong> Every write creates a table version, and old snapshots keep their data files alive. Storage grows even when the row count is flat.</li>



<li><strong>Orphan files never get cleaned.</strong> A killed write can leave files no snapshot references. They are invisible to every query and they bill every month forever.</li>
</ul>



<p class="wp-block-paragraph">None of that produces an alert. No threshold crossing, no failed job, no red panel. It is a slope, not a cliff, which is why teams hit the &#8220;Athena slow?&#8221; ticket eighteen months in with nothing to blame.</p>



<p class="wp-block-paragraph">Amazon S3 Tables exists to take those three jobs off you. That is a real pitch. The question is what you hand over to accept it.</p>



<h2 class="wp-block-heading">What S3 Tables actually runs for you</h2>



<p class="wp-block-paragraph">S3 Tables introduces a distinct bucket type, the table bucket, which holds tables as subresources rather than a prefix full of objects you manage. Inside it, S3 runs three maintenance operations, all on by default. They map one to one onto the standard Apache Iceberg maintenance routines, which tells you exactly what the do-it-yourself equivalent costs in effort.</p>



<h3 class="wp-block-heading">Compaction</h3>



<p class="wp-block-paragraph">Compaction merges small objects into larger ones and applies row-level deletes while it does. It is configured per table, with a target file size defaulting to 512MB and settable as low as 64MB. It reads Parquet, Avro and ORC, and writes Parquet by default.</p>



<p class="wp-block-paragraph">Two limits matter more than the defaults. Compaction does not support the <code>Fixed</code> data type, and it does not support brotli or lz4 compression. If your writer uses lz4 because someone benchmarked it years ago, you are paying for a managed service that cannot do the job you bought it for. Check the writer config before you migrate.</p>



<p class="wp-block-paragraph">Also worth internalizing: compaction runs on an automated schedule you do not control. There is no &#8220;compact now, I have a board report in an hour&#8221; button. Between a burst of small writes and the next pass, the table sits in exactly the state you were trying to avoid.</p>



<h3 class="wp-block-heading">Snapshot management, and the trap inside it</h3>



<p class="wp-block-paragraph">Snapshot management expires old table versions. Two per-table values govern it: a minimum number of snapshots to keep, and a maximum snapshot age. Both conditions must be satisfied before anything is removed, so the minimum count wins over the age setting on a quiet table.</p>



<p class="wp-block-paragraph">Here is the part that bites. Snapshot management does not honor retention set as Iceberg table properties. Configure a branch or tag based retention policy, or a retention in the table metadata longer than what you set through the maintenance API, and snapshot management is <em>disabled</em> for that table.</p>



<p class="wp-block-paragraph">Not overridden. Disabled. S3 stops expiring snapshots, you keep paying storage for every version, and nothing tells you. Someone runs an innocuous <code>ALTER TABLE ... SET TBLPROPERTIES</code> to tag a release for reproducibility, and months later the storage line has doubled on a table nobody thought was growing. If you adopt S3 Tables, make &#8220;nobody sets snapshot retention in table properties&#8221; a written rule and audit for it.</p>



<h3 class="wp-block-heading">Unreferenced file removal</h3>



<p class="wp-block-paragraph">This is orphan cleanup. It deletes data and metadata files no longer referenced by Iceberg metadata once they pass a retention window, configured as a number of days for unreferenced files and a number of days for non-current files.</p>



<p class="wp-block-paragraph">The structural detail is scope. Compaction and snapshot management are per table. Unreferenced file removal is configured <em>at the table bucket level only</em>. You cannot give a compliance-sensitive table a one day orphan window and a scratch table ten days in the same bucket. Different retention behavior means different table buckets, and that is a design-time decision.</p>



<p class="wp-block-paragraph">Editing any of this needs the maintenance configuration permissions, and automated runs surface in CloudTrail, which is your only real visibility into whether maintenance is happening.</p>



<pre class="wp-block-code"><code>s3tables:GetTableMaintenanceConfiguration
s3tables:PutTableMaintenanceConfiguration</code></pre>



<p class="wp-block-paragraph">Grant those to whoever owns the table, not to the pipeline role. A pipeline should never be able to switch off its own cleanup.</p>



<h2 class="wp-block-heading">S3 Tables: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> A high-frequency writer with no dedicated data platform engineer. Change data capture, clickstream, IoT telemetry, log tables: anything producing a steady drip of small files that nobody will schedule a compaction job for. AWS also positions table buckets as supporting substantially higher transaction rates than general purpose buckets, which is a vendor claim worth testing against your own write pattern rather than taking from a marketing page.</p>



<p class="wp-block-paragraph">It wins again when you want maintenance to be a property of the storage rather than a job in someone&#8217;s scheduler. A compaction job in Airflow can be deleted, break silently, or lose its owner when a contract ends. Maintenance that lives in the bucket does not.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Table buckets are not general purpose buckets, and the gap is wider than the name suggests.</p>



<ul class="wp-block-list">
<li>Public access policies and presigned URLs are not supported. If any workflow hands out object URLs, that pattern is gone.</li>



<li>Table bucket and table access policies are capped at 20 KB, which sounds generous until you are writing per-tenant conditions.</li>



<li>Replication has edges: replicas are read-only and compaction does not run on them, tables with tags or branches cannot be replicated, and a table upgraded from Iceberg V2 to V3 cannot be replicated afterward.</li>



<li>Your lifecycle rules, inventory reports, bucket-level scripts and storage class habits do not transfer directly.</li>



<li>Compaction timing is not yours. Where layout freshness matters within the hour, the scheduled model is a genuine constraint.</li>
</ul>



<h2 class="wp-block-heading">Iceberg on general purpose S3: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give this option its real case, because the &#8220;unmanaged&#8221; framing is out of date. Iceberg tables registered in the AWS Glue Data Catalog can have automatic table optimization enabled, covering compaction, snapshot retention and orphan file management. The optimizer watches partitions and starts compaction once file counts and sizes cross a threshold. The choice is no longer managed versus manual. It is closer to managed at the storage layer versus managed at the catalog layer.</p>



<p class="wp-block-paragraph"><strong>Where it wins.</strong></p>



<ul class="wp-block-list">
<li><strong>You control timing.</strong> Compact before a heavy reporting window or after a backfill, because you call the procedure.</li>



<li><strong>One bucket type, one mental model.</strong> Lifecycle rules, storage classes, bucket policies and VPC endpoint patterns keep working.</li>



<li><strong>Cheaper base storage rate.</strong> Table bucket storage carries a premium, and at volume that is not a rounding error.</li>



<li><strong>Exit is cheap.</strong> Iceberg and Parquet in a bucket you own. Moving catalogs is a metadata operation, not a data migration.</li>



<li><strong>Mixed workloads.</strong> Raw drops, partner exports and Iceberg tables can share one storage architecture.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> You own the failure mode. If nobody enables the optimizer or schedules the procedures, nothing runs and nothing complains. The work is not complicated, which is exactly the problem: it is easy enough that everyone assumes someone else did it.</p>



<p class="wp-block-paragraph">The manual path is three Iceberg Spark procedures on a schedule you own:</p>



<pre class="wp-block-code"><code>-- consolidate small data files
CALL catalog.system.rewrite_data_files(table =&gt; 'db.events')

-- drop old table versions
CALL catalog.system.expire_snapshots(table =&gt; 'db.events')

-- delete files no snapshot references
CALL catalog.system.remove_orphan_files(table =&gt; 'db.events')</code></pre>



<p class="wp-block-paragraph">Each accepts arguments for targets and retention that you should set deliberately rather than accept by default. Treat <code>remove_orphan_files</code> with care: it compares storage against metadata, and a short retention window can delete files a concurrent write is still committing.</p>



<p class="wp-block-paragraph">The other cost is compute. Those procedures run on Spark, so an EMR cluster or a Glue job, with its own IAM, network path, failure modes and bill. That is the real thing you buy out of with S3 Tables, and it is a bigger line than people estimate.</p>



<h2 class="wp-block-heading">The access path is a bigger difference than the maintenance</h2>



<p class="wp-block-paragraph">This is the part that generates support tickets, and most comparisons skip it. There are two REST catalog paths to an S3 table, with different authorization models.</p>



<ol class="wp-block-list">
<li><strong>Direct, through the S3 Tables Iceberg REST endpoint.</strong> Any Iceberg-compatible client connects. Authorization is IAM only, via identity-based policies or resource-based policies on the table and table bucket. AWS suggests this for self-managed or partner catalogs, or basic read and write against a single table bucket.</li>



<li><strong>Through the AWS Glue Data Catalog integration.</strong> Tables register as Data Catalog objects, visible to Athena, Redshift, EMR, Glue ETL and Firehose, with third-party clients reaching them through the Glue Iceberg REST endpoint. Authorization is IAM <em>and</em> Lake Formation. Both checks must pass.</li>
</ol>



<p class="wp-block-paragraph">That second point is where teams lose an afternoon. A PyIceberg client with a correct IAM policy still gets denied, because integrating with the Data Catalog registered the tables as Lake Formation resources and no Lake Formation grant exists. The IAM policy is not wrong. It is not sufficient. For external engines to receive credentials, Lake Formation also has to permit full table access for external engines, and a data lake administrator has to apply the grants.</p>



<p class="wp-block-paragraph">If you already run Lake Formation for fine-grained access control, this is a feature and column-level permissions come along for free. If you do not, you have added a governance service, an administrator role and a second permissions model to a project that was supposed to be about file sizes. Price that in honestly.</p>



<h2 class="wp-block-heading">How the two bills are shaped</h2>



<p class="wp-block-paragraph">Skip the per-gigabyte rates. They change, and the decision does not live there. Look at the shape instead.</p>



<p class="wp-block-paragraph"><strong>S3 Tables bills on four dimensions:</strong> storage, requests, a per-object monitoring fee for every object in the table bucket, and compaction charged both per object and per byte processed. The monitoring fee is the one people miss, because it scales with object <em>count</em>. The same workload that makes compaction expensive makes monitoring expensive, and both are driven by how many small files your writer produces.</p>



<p class="wp-block-paragraph"><strong>Iceberg on general purpose S3 bills on:</strong> storage, requests, and the compute you run maintenance on. That compute is lumpy and easy to underestimate, but it is a knob you control. A weekly compaction window on spot capacity costs very differently from continuous optimization.</p>



<p class="wp-block-paragraph">One lever moves both models: <strong>batch your writes</strong>. Committing every thirty seconds instead of every five minutes multiplies object count by ten, and object count feeds the monitoring fee, the compaction fee, the planning time and the Spark hours alike. Fixing the writer is almost always cheaper than choosing storage around a bad writer.</p>



<p class="wp-block-paragraph">Whichever you pick, make the cost visible. Cost allocation tags on the table bucket, a CloudWatch or Grafana panel on object counts, and CloudTrail on maintenance events will teach you more in a month than any spreadsheet estimate teaches you today.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for S3 Tables vs Iceberg on S3</h2>



<p class="wp-block-paragraph">Work through these for a specific table, not for your platform as a whole. Different tables can legitimately land in different places.</p>



<ol class="wp-block-list">
<li><strong>Count the commits per hour.</strong> A handful of commits a day from a batch job writing large files is not a small file problem, and managed compaction solves nothing. Stay on general purpose S3.</li>



<li><strong>Check the writer&#8217;s compression and types.</strong> lz4 or brotli, or a <code>Fixed</code> column, and S3 Tables compaction will not process those files. Fix the writer or stay put.</li>



<li><strong>Ask who owns maintenance today, by name.</strong> If it is a person who will still be here next year and a job that alerts on failure, do it yourself. If the answer is a shrug, that shrug is what you are paying to replace.</li>



<li><strong>Check for presigned URLs, public policies, lifecycle rules or large access policies on this data.</strong> Any yes rules out a table bucket for that table.</li>



<li><strong>Decide the governance model before the storage model.</strong> Want Lake Formation fine-grained control? The Glue Data Catalog path is a real benefit. Want IAM-only simplicity? Use direct access and keep Lake Formation out of it.</li>



<li><strong>Confirm nothing sets snapshot retention in table properties.</strong> If your pipeline or dbt project does, change it or accept that managed snapshot expiry will be silently off.</li>



<li><strong>Try the middle option first.</strong> Enable Glue Data Catalog table optimization on the existing table. Reversible, no data movement, and it answers &#8220;is automated maintenance enough&#8221; without committing you to a new bucket type.</li>
</ol>



<p class="wp-block-paragraph">That last step is the one I reach for first in almost every case. It delivers the outcome you actually want, maintenance that runs without a human, at the cost of a checkbox rather than a migration.</p>



<h2 class="wp-block-heading">Where teams get this wrong</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a storage migration.</strong> It is an operations and governance decision that happens to involve storage.</li>



<li><strong>Migrating the wrong table.</strong> Teams pick the biggest one. The right candidate is the highest commit frequency, which is often small.</li>



<li><strong>Assuming enabled means running.</strong> Table properties can disable snapshot expiry and unsupported compression can skip compaction. Verify in CloudTrail that operations actually fire.</li>



<li><strong>Forgetting concurrency.</strong> Iceberg uses optimistic concurrency, so compaction and your writes can conflict. Compaction retries on failure; your pipeline needs retry logic too, on both sides of this comparison.</li>



<li><strong>Skipping the exit plan.</strong> The data stays open, the bucket type does not. Write down how you would move back before you move in.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is S3 Tables just Iceberg with extra steps?</h3>



<p class="wp-block-paragraph">No. It is standard Apache Iceberg with maintenance run by the service and a purpose-built bucket type underneath. The table format is identical, so engines and data stay portable. What differs is who executes maintenance and which bucket-level features you give up to get it.</p>



<h3 class="wp-block-heading">Can I get managed maintenance without moving to a table bucket?</h3>



<p class="wp-block-paragraph">Yes. AWS Glue Data Catalog table optimization brings automatic compaction, snapshot retention and orphan file management to Iceberg tables in general purpose buckets. For most teams asking this question, that is the answer they actually wanted.</p>



<h3 class="wp-block-heading">Does compaction break time travel?</h3>



<p class="wp-block-paragraph">Compaction writes a new snapshot, so history survives. Snapshot management is what removes time travel, because expiring a snapshot deletes its metadata and can delete the associated data files. If you rely on travelling to a specific point, set the minimum snapshot count and maximum snapshot age to cover that window deliberately.</p>



<h3 class="wp-block-heading">Why does my PyIceberg client get access denied on S3 Tables?</h3>



<p class="wp-block-paragraph">Usually because you are going through the Glue Data Catalog integration, where both IAM and Lake Formation must authorize the request and only the IAM half is configured. Either add the Lake Formation grants and enable external engine access, or switch to the S3 Tables Iceberg REST endpoint for direct IAM-only access.</p>



<h3 class="wp-block-heading">Is S3 Tables more expensive than regular S3?</h3>



<p class="wp-block-paragraph">The storage rate carries a premium, plus a per-object monitoring fee and compaction charges. Against that, you remove the compute you were spending on maintenance. Whether the total is higher depends almost entirely on object count, so model it from your file counts rather than your terabytes.</p>



<h3 class="wp-block-heading">Can I disable compaction and control it myself?</h3>



<p class="wp-block-paragraph">Yes, compaction can be disabled per table through the maintenance configuration API, which also stops the associated charges. Reasonable if you already have a tuned compaction job, though it undercuts most of the reason to be on a table bucket.</p>



<h3 class="wp-block-heading">Which should a small team pick?</h3>



<p class="wp-block-paragraph">A small team with one high-churn streaming table and no data platform engineer is the clearest fit for S3 Tables. A small team running a nightly batch load into a few large tables should stay on general purpose S3 and turn on Glue table optimization if maintenance is the worry.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables vs Iceberg on S3 is not really a performance comparison, and not really a cost comparison either. Both store the same Parquet under the same open table format, and either performs well when the layout is maintained.</p>



<p class="wp-block-paragraph">What you are choosing is where responsibility for compaction, snapshot expiry and orphan cleanup lives, and how confident you are that it will still be happening in two years when the person who set it up has moved on. Low confidence and a high commit rate means managed Iceberg earns its premium. A real owner, a real schedule and large batch writes means you are paying for a problem you do not have.</p>



<p class="wp-block-paragraph">Either way, check the two silent failures before you ship: table properties that disable snapshot expiry, and a writer whose compression codec compaction cannot read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or fixing a table that already rotted?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for teams running Iceberg on AWS:</p>



<ul class="wp-block-list">
<li>Auditing Iceberg tables for small file growth, snapshot bloat and orphan files, with a before-and-after on planning time and storage</li>



<li>Modelling S3 Tables against your real object counts and commit frequency, so the decision comes from your numbers rather than a pricing page example</li>



<li>Setting up AWS Glue Data Catalog table optimization as the low-risk middle path, including the IAM and network configuration it needs</li>



<li>Untangling Lake Formation and IAM when external Iceberg clients get denied through the Glue REST endpoint</li>



<li>Tuning the writer side, where the actual fix usually lives: commit batching, target file sizes, partition layout and compression choices</li>



<li>Building monitoring that makes maintenance visible, so degradation shows up on a dashboard instead of in a &#8220;why is Athena slow&#8221; ticket</li>
</ul>



<p class="wp-block-paragraph">Send me a table&#8217;s manifest summary, a Glue optimizer config, or the planning stage of a slow query, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>S3 Tables Compaction in Production: The Cost of Convenience</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 22 Oct 2026 06:02:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Query Performance]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Write Amplification]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=773</guid>

					<description><![CDATA[<p>Managed Iceberg compaction in S3 Tables removes real operational work, but it will not tell you when it stops. A practical look at the maintenance jobs that fail silently, the retention cascade that keeps superseded data on your bill for two weeks, how compaction charges actually accumulate, and the handful of knobs worth touching.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The table bucket had been in production a couple of months. Ingest volume was flat, retention said ninety days, and the storage line on the bill kept climbing anyway. Compaction was enabled. The console showed nothing wrong. Someone had already checked twice.</p>



<p class="wp-block-paragraph">The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.</p>



<p class="wp-block-paragraph">That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.</p>



<h2 class="wp-block-heading">What managed compaction actually does</h2>



<p class="wp-block-paragraph">S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.</p>



<p class="wp-block-paragraph">Compaction is the Iceberg <code>rewriteDataFiles</code> routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table&#8217;s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.</p>



<p class="wp-block-paragraph">There are four strategies. <code>auto</code> is the default and picks for you: sort order defined in metadata means <code>sort</code>, otherwise <code>binpack</code>. <code>binpack</code> packs small files into bigger ones. <code>sort</code> orders rows hierarchically by your chosen columns so engines prune more files. <code>z-order</code> interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.</p>



<p class="wp-block-paragraph">Both <code>sort</code> and <code>z-order</code> need a <code>sort_order</code> in your Iceberg table properties and the <code>s3tables:GetTableData</code> permission to set. They also cost more per byte processed than <code>binpack</code>, because they do more work.</p>



<h2 class="wp-block-heading">Failure family one: maintenance that fails without telling anyone</h2>



<p class="wp-block-paragraph">This is the one worth reading the post for. Maintenance jobs have four states: <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, and <code>Not_Yet_Run</code>. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.</p>



<pre class="wp-block-code"><code>aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table</code></pre>



<p class="wp-block-paragraph">The response carries a block for each of <code>icebergCompaction</code>, <code>icebergSnapshotManagement</code>, and <code>icebergUnreferencedFileRemoval</code>, each with its own status, a <code>lastRunTimestamp</code>, and a <code>failureMessage</code>. The three are independent. Compaction reporting <code>Successful</code> tells you nothing about whether the other two are running.</p>



<p class="wp-block-paragraph">The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.</p>



<h3 class="wp-block-heading">Tags and branches disable snapshot expiry for the whole table</h3>



<p class="wp-block-paragraph">If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.</p>



<p class="wp-block-paragraph">Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month&#8217;s bill rather than in their terminal.</p>



<h3 class="wp-block-heading">Iceberg retention properties conflict with the managed config</h3>



<p class="wp-block-paragraph">If <code>history.expire.max-snapshot-age-ms</code> or <code>history.expire.min-snapshots-to-keep</code> is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in <code>metadata.json</code>. It refuses instead.</p>



<p class="wp-block-paragraph">This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.</p>



<pre class="wp-block-code"><code>ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');</code></pre>



<h3 class="wp-block-heading">gc.enabled set to false</h3>



<p class="wp-block-paragraph">Setting the Iceberg <code>gc.enabled</code> table property to <code>false</code> stops both snapshot management and unreferenced file removal. Leave it unset or explicitly <code>true</code>. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.</p>



<h3 class="wp-block-heading">KMS key policy missing the maintenance principal</h3>



<p class="wp-block-paragraph">If the table is encrypted with a customer managed key, that key&#8217;s policy has to grant access to the service principal <code>maintenance.s3tables.amazonaws.com</code>. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.</p>



<h2 class="wp-block-heading">Failure family two: the storage tail you keep paying for</h2>



<p class="wp-block-paragraph">Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.</p>



<p class="wp-block-paragraph">When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from <code>minSnapshotsToKeep</code> (default 1) and <code>maxSnapshotAgeHours</code> (default 120). Both conditions must be satisfied before a snapshot goes.</p>



<p class="wp-block-paragraph">When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for <code>nonCurrentDays</code> (default 10), then are deleted permanently.</p>



<p class="wp-block-paragraph">Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through <code>unreferencedDays</code> (default 3) into the same noncurrent window.</p>



<p class="wp-block-paragraph">Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.</p>



<pre class="wp-block-code"><code># Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'</code></pre>



<p class="wp-block-paragraph">Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.</p>



<h2 class="wp-block-heading">Failure family three: write amplification and the compaction charge</h2>



<p class="wp-block-paragraph">S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.</p>



<p class="wp-block-paragraph">The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.</p>



<p class="wp-block-paragraph">People reach for <code>targetFileSizeMB</code> first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.</p>



<p class="wp-block-paragraph">The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.</p>



<p class="wp-block-paragraph">Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.</p>



<h2 class="wp-block-heading">Failure family four: files compaction quietly skips</h2>



<p class="wp-block-paragraph">Compaction supports Parquet, Avro, and ORC. It does not support the <code>Fixed</code> data type, nor the <code>brotli</code> or <code>lz4</code> compression codecs.</p>



<p class="wp-block-paragraph">That last one matters because <code>lz4</code> is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.</p>



<p class="wp-block-paragraph">Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the <code>write.format.default</code> table property to <code>avro</code> or <code>orc</code>. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.</p>



<h2 class="wp-block-heading">Failure family five: compaction and your writers fighting over the table</h2>



<p class="wp-block-paragraph">Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.</p>



<p class="wp-block-paragraph">When compaction loses you see <code>Failed</code> with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between <code>Failed</code> and <code>Successful</code>.</p>



<p class="wp-block-paragraph">When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.</p>



<h2 class="wp-block-heading">Turning managed compaction off, and when that is right</h2>



<p class="wp-block-paragraph">Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.</p>



<p class="wp-block-paragraph">The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.</p>



<p class="wp-block-paragraph">Disabling is a per-table change, and it does not require zeroing the settings.</p>



<pre class="wp-block-code"><code>aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'</code></pre>



<p class="wp-block-paragraph">From there you run maintenance yourself. Trino exposes an <code>optimize</code> procedure through <code>ALTER TABLE ... EXECUTE</code>, and Spark has the standard Iceberg <code>rewrite_data_files</code> procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.</p>



<p class="wp-block-paragraph">Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting S3 Tables compaction and maintenance</h2>



<p class="wp-block-paragraph">Work through these in order. The first step resolves more cases than the rest combined.</p>



<ol class="wp-block-list">
<li><strong>Check all three job statuses, not one.</strong> Run <code>get-table-maintenance-job-status</code> and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.</li>

<li><strong>Read the failure message literally.</strong> It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means <code>history.expire.*</code> is set.</li>

<li><strong>Status is <code>Disabled</code> or <code>Not_Yet_Run</code> unexpectedly.</strong> Someone replaced the configuration. Confirm with <code>get-table-maintenance-configuration</code>, and remember <code>put-table-maintenance-configuration</code> replaces rather than merges.</li>

<li><strong>Compaction runs but file count stays high.</strong> Check codec and data types against the unsupported list. Files using <code>brotli</code> or <code>lz4</code>, or containing the <code>Fixed</code> type, are outside what compaction touches.</li>

<li><strong>Insufficient-access failures on an encrypted table.</strong> Check the KMS key policy grants the <code>maintenance.s3tables.amazonaws.com</code> service principal, and that the key is enabled and still exists.</li>

<li><strong>Storage still growing with everything green.</strong> Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.</li>
</ol>



<p class="wp-block-paragraph">For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer &#8220;when did this start&#8221; rather than &#8220;what is broken now&#8221;.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the three maintenance jobs as one thing.</strong> They fail independently and for different reasons. &#8220;Maintenance is enabled&#8221; is not a status.</li>

<li><strong>Creating branches or tags on a managed table.</strong> A normal Iceberg workflow that switches off snapshot expiry for the whole table.</li>

<li><strong>Tuning <code>targetFileSizeMB</code> to reduce cost.</strong> It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.</li>

<li><strong>Setting <code>sort</code> or <code>z-order</code> without a sort order defined.</strong> The strategy needs <code>sort_order</code> in table properties and the <code>s3tables:GetTableData</code> permission, and it costs more per byte than <code>binpack</code>.</li>

<li><strong>Running write pipelines with no commit retry.</strong> Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.</li>
</ul>



<h2 class="wp-block-heading">Best practices for running S3 Tables compaction in production</h2>



<ul class="wp-block-list">
<li><strong>Poll job status on a schedule and alert on it.</strong> A small scheduled function calling <code>get-table-maintenance-job-status</code> across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on <code>Failed</code> and on a stale <code>lastRunTimestamp</code>, on whatever dashboard you already watch, Grafana or CloudWatch.</li>

<li><strong>Make maintenance config part of your IaC.</strong> These are settings someone can change with one CLI call and no review. Codify them so drift is visible.</li>

<li><strong>Set retention windows from a real requirement.</strong> Ask how far back anyone needs to time travel, then set <code>maxSnapshotAgeHours</code> and <code>nonCurrentDays</code> to that plus a margin, rather than accepting defaults by omission.</li>

<li><strong>Batch on the write side before tuning anything else.</strong> Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.</li>

<li><strong>Choose the strategy from the query pattern.</strong> One dominant filter column suggests <code>sort</code>. Several equally weighted predicates suggest <code>z-order</code>. Neither suggests leaving it on <code>auto</code>.</li>

<li><strong>Track the compaction billing dimension separately.</strong> It moves independently of storage and it is the earliest signal that a writer&#8217;s behaviour has changed.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I know if S3 Tables compaction is actually running?</h3>



<p class="wp-block-paragraph">Call <code>get-table-maintenance-job-status</code> and read the <code>icebergCompaction</code> block. It returns <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, or <code>Not_Yet_Run</code>, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.</p>



<h3 class="wp-block-heading">Why is my S3 Tables storage growing when compaction says it succeeded?</h3>



<p class="wp-block-paragraph">Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.</p>



<h3 class="wp-block-heading">Can I run my own compaction alongside the managed one?</h3>



<p class="wp-block-paragraph">You can, but it is a poor idea. Both are writers competing under Iceberg&#8217;s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.</p>



<h3 class="wp-block-heading">Does disabling compaction stop all S3 Tables charges?</h3>



<p class="wp-block-paragraph">No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.</p>



<h3 class="wp-block-heading">Is sort or z-order compaction worth the extra cost?</h3>



<p class="wp-block-paragraph">It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour <code>sort</code>. Several dimensions in varying combinations favour <code>z-order</code>. Queries that mostly scan everything justify neither.</p>



<h3 class="wp-block-heading">Can I recover data deleted by S3 Tables maintenance?</h3>



<p class="wp-block-paragraph">Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.</p>



<p class="wp-block-paragraph">What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on <code>get-table-maintenance-job-status</code> that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.</p>



<h2 class="wp-block-heading">Need help with Iceberg tables on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots</li>

<li>Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana</li>

<li>Working out where a bucket&#8217;s storage growth actually comes from, and setting retention windows against a real requirement</li>

<li>Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them</li>

<li>Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick</li>

<li>Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance</li>
</ul>



<p class="wp-block-paragraph">If something is not adding up, send me the output of <code>get-table-maintenance-job-status</code> for the table and a rough description of your write pattern. That is usually enough to say what is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</title>
		<link>https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 06:06:48 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=500</guid>

					<description><![CDATA[<p>S3 holds the documents, Iceberg holds the rows, and everything works right up until a number disagrees with its source PDF. Combining structured and unstructured data on AWS is an identity problem, not a storage one. Here are the five failure families that cause it, why each stays invisible until it isn't, and the ingestion order that avoids most of them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;the contract value in the dashboard is wrong.&#8221; It wasn&#8217;t. The number in the warehouse table was right for the version of the PDF that had been extracted eleven months earlier. Someone had uploaded an amended contract to the same S3 prefix, the extraction job had picked it up, written a new row, and nothing downstream knew which of the two rows was current. Both were true. Only one was correct.</p>



<p class="wp-block-paragraph">That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.</p>



<p class="wp-block-paragraph">The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.</p>



<h2 class="wp-block-heading">What this post covers</h2>



<p class="wp-block-paragraph">Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn&#8217;t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.</p>



<p class="wp-block-paragraph">Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: there is no join key, only a filename</h2>



<p class="wp-block-paragraph">Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.</p>



<p class="wp-block-paragraph">Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.</p>



<h3 class="wp-block-heading">What to use instead</h3>



<p class="wp-block-paragraph">Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.</p>



<p class="wp-block-paragraph">Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for &#8220;this exact bytes-on-disk artefact.&#8221; If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.</p>



<p class="wp-block-paragraph">S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.</p>



<h2 class="wp-block-heading">Failure family two: extraction confidence gets flattened into a fact</h2>



<p class="wp-block-paragraph">Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.</p>



<p class="wp-block-paragraph">Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn&#8217;t go away. You just stopped being able to see it.</p>



<p class="wp-block-paragraph">Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always &#8220;which extractor produced these,&#8221; and without that column the answer is a week of archaeology.</p>



<pre class="wp-block-code"><code>-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);</code></pre>



<p class="wp-block-paragraph">The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.</p>



<h2 class="wp-block-heading">Failure family three: two catalogs, two permission models</h2>



<p class="wp-block-paragraph">This is the one that bites teams who did everything else right.</p>



<p class="wp-block-paragraph">Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.</p>



<p class="wp-block-paragraph">The result is predictable and unpleasant. An analyst who is correctly denied the <code>salary</code> column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.</p>



<p class="wp-block-paragraph">There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:</p>



<ul class="wp-block-list">

<li>Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.</li>


<li>Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.</li>


<li>Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.</li>


<li>Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.</li>

</ul>



<p class="wp-block-paragraph">Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with <code>.metadata.json</code> appended. That file is where classification, tenant, document ID and effective dates belong.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}</code></pre>



<p class="wp-block-paragraph">Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.</p>



<h2 class="wp-block-heading">Failure family four: index decisions you cannot take back</h2>



<p class="wp-block-paragraph">S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.</p>



<p class="wp-block-paragraph">By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.</p>



<p class="wp-block-paragraph">If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.</p>



<pre class="wp-block-code"><code># Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'</code></pre>



<p class="wp-block-paragraph">The dimension has to match your embedding model&#8217;s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.</p>



<p class="wp-block-paragraph">Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing rewrites history</h2>



<p class="wp-block-paragraph">You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter&#8217;s board pack change.</p>



<p class="wp-block-paragraph">This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.</p>



<p class="wp-block-paragraph">Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and &#8220;why did this number move&#8221; has an answer you can query instead of guess at.</p>



<p class="wp-block-paragraph">Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.</p>



<h2 class="wp-block-heading">A shape that works for combining structured and unstructured data on AWS</h2>



<p class="wp-block-paragraph">Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.</p>



<ol class="wp-block-list">

<li><strong>Land raw, enable versioning, enable S3 Metadata.</strong> Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.</li>


<li><strong>Mint the document ID at landing.</strong> Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.</li>


<li><strong>Classify before you extract.</strong> Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.</li>


<li><strong>Extract to a staging table, not straight to the curated table.</strong> Staging gives you somewhere to run validation before anything touches a table an analyst can query.</li>


<li><strong>Merge into Iceberg on document identity, carrying confidence and provenance columns.</strong> Never merge on object key.</li>


<li><strong>Embed separately, keyed by the same document ID.</strong> The vector plane and the table plane are different systems with different refresh rates. Treat them as such.</li>


<li><strong>Join at query time in Athena.</strong> Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.</li>

</ol>



<p class="wp-block-paragraph">Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Athena says the principal has no privilege on the resource</h3>



<p class="wp-block-paragraph">This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.</p>



<h3 class="wp-block-heading">Queries against metadata tables fail with a syntax error</h3>



<p class="wp-block-paragraph">Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.</p>



<h3 class="wp-block-heading">Knowledge base sync fails partway with validation errors</h3>



<p class="wp-block-paragraph">Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.</p>



<h3 class="wp-block-heading">Retrieval returns documents the caller should not see</h3>



<p class="wp-block-paragraph">Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.</p>



<h3 class="wp-block-heading">Athena gets slower week over week on a table that isn&#8217;t growing much</h3>



<p class="wp-block-paragraph">Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Joining on the S3 object key because it was there and it worked in the prototype.</li>


<li>Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.</li>


<li>Applying a confidence threshold and discarding the score.</li>


<li>Creating the vector index before deciding on the embedding model.</li>


<li>Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.</li>


<li>Reprocessing in place, with no record of which extractor produced which value.</li>


<li>Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.</li>


<li>Confidence and provenance columns are not optional. They are the difference between debugging and guessing.</li>


<li>Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.</li>


<li>Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.</li>


<li>Enforce classification in all three planes independently. There is no single choke point.</li>


<li>Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.</li>


<li>Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a lakehouse to combine structured and unstructured data on AWS?</h3>



<p class="wp-block-paragraph">No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.</p>



<h3 class="wp-block-heading">Should extraction output live in the same table as the structured source data?</h3>



<p class="wp-block-paragraph">Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.</p>



<h3 class="wp-block-heading">Is S3 Vectors a replacement for OpenSearch?</h3>



<p class="wp-block-paragraph">For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.</p>



<h3 class="wp-block-heading">How do I keep the vector index in sync with the documents?</h3>



<p class="wp-block-paragraph">Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because &#8220;mostly in sync&#8221; degrades quietly.</p>



<h3 class="wp-block-heading">What breaks first when this architecture is done badly?</h3>



<p class="wp-block-paragraph">Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.</p>



<h3 class="wp-block-heading">Can I use Textract instead of Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.</p>



<h3 class="wp-block-heading">Does Lake Formation govern my documents as well as my tables?</h3>



<p class="wp-block-paragraph">It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.</p>



<p class="wp-block-paragraph">Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help wiring documents and tables together?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:</p>



<ul class="wp-block-list">

<li>Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.</li>


<li>Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.</li>


<li>Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.</li>


<li>Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.</li>


<li>Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.</li>


<li>Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.</li>

</ul>



<p class="wp-block-paragraph">Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
