<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Iceberg REST Catalog | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/iceberg-rest-catalog/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/iceberg-rest-catalog/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 14 Nov 2026 07:01:31 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Iceberg REST Catalog | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/iceberg-rest-catalog/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 14 Nov 2026 07:01:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=683</guid>

					<description><![CDATA[<p>Managed Iceberg is not a storage upgrade, it is a decision about who owns compaction, snapshot expiry and orphan cleanup. A practical comparison of S3 Tables against Iceberg in general purpose S3 buckets, covering the maintenance you actually get, the Lake Formation trap in the access path, how the two bills are shaped, and a seven-step decision procedure you can run against a single table.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The dashboard that used to load in eight seconds now takes ninety. Nobody changed the query, nobody changed the schema, and the table is barely bigger than it was last quarter. Someone opens a ticket titled &#8220;Athena slow?&#8221; and it sits there, because there is no error to point at.</p>



<p class="wp-block-paragraph">What changed is the file count. A streaming job has been writing small Parquet files every few minutes for months, and each one is another entry the query planner opens before it can plan anything. The data is fine. The layout rotted.</p>



<p class="wp-block-paragraph">That is the problem underneath the whole S3 Tables vs Iceberg on S3 argument. It is not a storage question. It is a question about who owns table maintenance, and what happens when the answer is nobody. This post covers what S3 Tables actually runs for you, where each option wins, why the access path matters more than most comparisons admit, how the two bills are shaped, and a decision procedure you can apply to a specific table.</p>



<h2 class="wp-block-heading">The failure that doesn&#8217;t announce itself</h2>



<p class="wp-block-paragraph">Iceberg on general purpose S3 works beautifully on day one. You write Parquet, commit a snapshot, Athena reads it. Nothing about that first week tells you it needs upkeep. Three things degrade quietly afterward.</p>



<ul class="wp-block-list">
<li><strong>Small files accumulate.</strong> Planning cost scales with how many files the planner considers, not with how much data you actually scan.</li>



<li><strong>Snapshots pile up.</strong> Every write creates a table version, and old snapshots keep their data files alive. Storage grows even when the row count is flat.</li>



<li><strong>Orphan files never get cleaned.</strong> A killed write can leave files no snapshot references. They are invisible to every query and they bill every month forever.</li>
</ul>



<p class="wp-block-paragraph">None of that produces an alert. No threshold crossing, no failed job, no red panel. It is a slope, not a cliff, which is why teams hit the &#8220;Athena slow?&#8221; ticket eighteen months in with nothing to blame.</p>



<p class="wp-block-paragraph">Amazon S3 Tables exists to take those three jobs off you. That is a real pitch. The question is what you hand over to accept it.</p>



<h2 class="wp-block-heading">What S3 Tables actually runs for you</h2>



<p class="wp-block-paragraph">S3 Tables introduces a distinct bucket type, the table bucket, which holds tables as subresources rather than a prefix full of objects you manage. Inside it, S3 runs three maintenance operations, all on by default. They map one to one onto the standard Apache Iceberg maintenance routines, which tells you exactly what the do-it-yourself equivalent costs in effort.</p>



<h3 class="wp-block-heading">Compaction</h3>



<p class="wp-block-paragraph">Compaction merges small objects into larger ones and applies row-level deletes while it does. It is configured per table, with a target file size defaulting to 512MB and settable as low as 64MB. It reads Parquet, Avro and ORC, and writes Parquet by default.</p>



<p class="wp-block-paragraph">Two limits matter more than the defaults. Compaction does not support the <code>Fixed</code> data type, and it does not support brotli or lz4 compression. If your writer uses lz4 because someone benchmarked it years ago, you are paying for a managed service that cannot do the job you bought it for. Check the writer config before you migrate.</p>



<p class="wp-block-paragraph">Also worth internalizing: compaction runs on an automated schedule you do not control. There is no &#8220;compact now, I have a board report in an hour&#8221; button. Between a burst of small writes and the next pass, the table sits in exactly the state you were trying to avoid.</p>



<h3 class="wp-block-heading">Snapshot management, and the trap inside it</h3>



<p class="wp-block-paragraph">Snapshot management expires old table versions. Two per-table values govern it: a minimum number of snapshots to keep, and a maximum snapshot age. Both conditions must be satisfied before anything is removed, so the minimum count wins over the age setting on a quiet table.</p>



<p class="wp-block-paragraph">Here is the part that bites. Snapshot management does not honor retention set as Iceberg table properties. Configure a branch or tag based retention policy, or a retention in the table metadata longer than what you set through the maintenance API, and snapshot management is <em>disabled</em> for that table.</p>



<p class="wp-block-paragraph">Not overridden. Disabled. S3 stops expiring snapshots, you keep paying storage for every version, and nothing tells you. Someone runs an innocuous <code>ALTER TABLE ... SET TBLPROPERTIES</code> to tag a release for reproducibility, and months later the storage line has doubled on a table nobody thought was growing. If you adopt S3 Tables, make &#8220;nobody sets snapshot retention in table properties&#8221; a written rule and audit for it.</p>



<h3 class="wp-block-heading">Unreferenced file removal</h3>



<p class="wp-block-paragraph">This is orphan cleanup. It deletes data and metadata files no longer referenced by Iceberg metadata once they pass a retention window, configured as a number of days for unreferenced files and a number of days for non-current files.</p>



<p class="wp-block-paragraph">The structural detail is scope. Compaction and snapshot management are per table. Unreferenced file removal is configured <em>at the table bucket level only</em>. You cannot give a compliance-sensitive table a one day orphan window and a scratch table ten days in the same bucket. Different retention behavior means different table buckets, and that is a design-time decision.</p>



<p class="wp-block-paragraph">Editing any of this needs the maintenance configuration permissions, and automated runs surface in CloudTrail, which is your only real visibility into whether maintenance is happening.</p>



<pre class="wp-block-code"><code>s3tables:GetTableMaintenanceConfiguration
s3tables:PutTableMaintenanceConfiguration</code></pre>



<p class="wp-block-paragraph">Grant those to whoever owns the table, not to the pipeline role. A pipeline should never be able to switch off its own cleanup.</p>



<h2 class="wp-block-heading">S3 Tables: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> A high-frequency writer with no dedicated data platform engineer. Change data capture, clickstream, IoT telemetry, log tables: anything producing a steady drip of small files that nobody will schedule a compaction job for. AWS also positions table buckets as supporting substantially higher transaction rates than general purpose buckets, which is a vendor claim worth testing against your own write pattern rather than taking from a marketing page.</p>



<p class="wp-block-paragraph">It wins again when you want maintenance to be a property of the storage rather than a job in someone&#8217;s scheduler. A compaction job in Airflow can be deleted, break silently, or lose its owner when a contract ends. Maintenance that lives in the bucket does not.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Table buckets are not general purpose buckets, and the gap is wider than the name suggests.</p>



<ul class="wp-block-list">
<li>Public access policies and presigned URLs are not supported. If any workflow hands out object URLs, that pattern is gone.</li>



<li>Table bucket and table access policies are capped at 20 KB, which sounds generous until you are writing per-tenant conditions.</li>



<li>Replication has edges: replicas are read-only and compaction does not run on them, tables with tags or branches cannot be replicated, and a table upgraded from Iceberg V2 to V3 cannot be replicated afterward.</li>



<li>Your lifecycle rules, inventory reports, bucket-level scripts and storage class habits do not transfer directly.</li>



<li>Compaction timing is not yours. Where layout freshness matters within the hour, the scheduled model is a genuine constraint.</li>
</ul>



<h2 class="wp-block-heading">Iceberg on general purpose S3: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give this option its real case, because the &#8220;unmanaged&#8221; framing is out of date. Iceberg tables registered in the AWS Glue Data Catalog can have automatic table optimization enabled, covering compaction, snapshot retention and orphan file management. The optimizer watches partitions and starts compaction once file counts and sizes cross a threshold. The choice is no longer managed versus manual. It is closer to managed at the storage layer versus managed at the catalog layer.</p>



<p class="wp-block-paragraph"><strong>Where it wins.</strong></p>



<ul class="wp-block-list">
<li><strong>You control timing.</strong> Compact before a heavy reporting window or after a backfill, because you call the procedure.</li>



<li><strong>One bucket type, one mental model.</strong> Lifecycle rules, storage classes, bucket policies and VPC endpoint patterns keep working.</li>



<li><strong>Cheaper base storage rate.</strong> Table bucket storage carries a premium, and at volume that is not a rounding error.</li>



<li><strong>Exit is cheap.</strong> Iceberg and Parquet in a bucket you own. Moving catalogs is a metadata operation, not a data migration.</li>



<li><strong>Mixed workloads.</strong> Raw drops, partner exports and Iceberg tables can share one storage architecture.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> You own the failure mode. If nobody enables the optimizer or schedules the procedures, nothing runs and nothing complains. The work is not complicated, which is exactly the problem: it is easy enough that everyone assumes someone else did it.</p>



<p class="wp-block-paragraph">The manual path is three Iceberg Spark procedures on a schedule you own:</p>



<pre class="wp-block-code"><code>-- consolidate small data files
CALL catalog.system.rewrite_data_files(table =&gt; 'db.events')

-- drop old table versions
CALL catalog.system.expire_snapshots(table =&gt; 'db.events')

-- delete files no snapshot references
CALL catalog.system.remove_orphan_files(table =&gt; 'db.events')</code></pre>



<p class="wp-block-paragraph">Each accepts arguments for targets and retention that you should set deliberately rather than accept by default. Treat <code>remove_orphan_files</code> with care: it compares storage against metadata, and a short retention window can delete files a concurrent write is still committing.</p>



<p class="wp-block-paragraph">The other cost is compute. Those procedures run on Spark, so an EMR cluster or a Glue job, with its own IAM, network path, failure modes and bill. That is the real thing you buy out of with S3 Tables, and it is a bigger line than people estimate.</p>



<h2 class="wp-block-heading">The access path is a bigger difference than the maintenance</h2>



<p class="wp-block-paragraph">This is the part that generates support tickets, and most comparisons skip it. There are two REST catalog paths to an S3 table, with different authorization models.</p>



<ol class="wp-block-list">
<li><strong>Direct, through the S3 Tables Iceberg REST endpoint.</strong> Any Iceberg-compatible client connects. Authorization is IAM only, via identity-based policies or resource-based policies on the table and table bucket. AWS suggests this for self-managed or partner catalogs, or basic read and write against a single table bucket.</li>



<li><strong>Through the AWS Glue Data Catalog integration.</strong> Tables register as Data Catalog objects, visible to Athena, Redshift, EMR, Glue ETL and Firehose, with third-party clients reaching them through the Glue Iceberg REST endpoint. Authorization is IAM <em>and</em> Lake Formation. Both checks must pass.</li>
</ol>



<p class="wp-block-paragraph">That second point is where teams lose an afternoon. A PyIceberg client with a correct IAM policy still gets denied, because integrating with the Data Catalog registered the tables as Lake Formation resources and no Lake Formation grant exists. The IAM policy is not wrong. It is not sufficient. For external engines to receive credentials, Lake Formation also has to permit full table access for external engines, and a data lake administrator has to apply the grants.</p>



<p class="wp-block-paragraph">If you already run Lake Formation for fine-grained access control, this is a feature and column-level permissions come along for free. If you do not, you have added a governance service, an administrator role and a second permissions model to a project that was supposed to be about file sizes. Price that in honestly.</p>



<h2 class="wp-block-heading">How the two bills are shaped</h2>



<p class="wp-block-paragraph">Skip the per-gigabyte rates. They change, and the decision does not live there. Look at the shape instead.</p>



<p class="wp-block-paragraph"><strong>S3 Tables bills on four dimensions:</strong> storage, requests, a per-object monitoring fee for every object in the table bucket, and compaction charged both per object and per byte processed. The monitoring fee is the one people miss, because it scales with object <em>count</em>. The same workload that makes compaction expensive makes monitoring expensive, and both are driven by how many small files your writer produces.</p>



<p class="wp-block-paragraph"><strong>Iceberg on general purpose S3 bills on:</strong> storage, requests, and the compute you run maintenance on. That compute is lumpy and easy to underestimate, but it is a knob you control. A weekly compaction window on spot capacity costs very differently from continuous optimization.</p>



<p class="wp-block-paragraph">One lever moves both models: <strong>batch your writes</strong>. Committing every thirty seconds instead of every five minutes multiplies object count by ten, and object count feeds the monitoring fee, the compaction fee, the planning time and the Spark hours alike. Fixing the writer is almost always cheaper than choosing storage around a bad writer.</p>



<p class="wp-block-paragraph">Whichever you pick, make the cost visible. Cost allocation tags on the table bucket, a CloudWatch or Grafana panel on object counts, and CloudTrail on maintenance events will teach you more in a month than any spreadsheet estimate teaches you today.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for S3 Tables vs Iceberg on S3</h2>



<p class="wp-block-paragraph">Work through these for a specific table, not for your platform as a whole. Different tables can legitimately land in different places.</p>



<ol class="wp-block-list">
<li><strong>Count the commits per hour.</strong> A handful of commits a day from a batch job writing large files is not a small file problem, and managed compaction solves nothing. Stay on general purpose S3.</li>



<li><strong>Check the writer&#8217;s compression and types.</strong> lz4 or brotli, or a <code>Fixed</code> column, and S3 Tables compaction will not process those files. Fix the writer or stay put.</li>



<li><strong>Ask who owns maintenance today, by name.</strong> If it is a person who will still be here next year and a job that alerts on failure, do it yourself. If the answer is a shrug, that shrug is what you are paying to replace.</li>



<li><strong>Check for presigned URLs, public policies, lifecycle rules or large access policies on this data.</strong> Any yes rules out a table bucket for that table.</li>



<li><strong>Decide the governance model before the storage model.</strong> Want Lake Formation fine-grained control? The Glue Data Catalog path is a real benefit. Want IAM-only simplicity? Use direct access and keep Lake Formation out of it.</li>



<li><strong>Confirm nothing sets snapshot retention in table properties.</strong> If your pipeline or dbt project does, change it or accept that managed snapshot expiry will be silently off.</li>



<li><strong>Try the middle option first.</strong> Enable Glue Data Catalog table optimization on the existing table. Reversible, no data movement, and it answers &#8220;is automated maintenance enough&#8221; without committing you to a new bucket type.</li>
</ol>



<p class="wp-block-paragraph">That last step is the one I reach for first in almost every case. It delivers the outcome you actually want, maintenance that runs without a human, at the cost of a checkbox rather than a migration.</p>



<h2 class="wp-block-heading">Where teams get this wrong</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a storage migration.</strong> It is an operations and governance decision that happens to involve storage.</li>



<li><strong>Migrating the wrong table.</strong> Teams pick the biggest one. The right candidate is the highest commit frequency, which is often small.</li>



<li><strong>Assuming enabled means running.</strong> Table properties can disable snapshot expiry and unsupported compression can skip compaction. Verify in CloudTrail that operations actually fire.</li>



<li><strong>Forgetting concurrency.</strong> Iceberg uses optimistic concurrency, so compaction and your writes can conflict. Compaction retries on failure; your pipeline needs retry logic too, on both sides of this comparison.</li>



<li><strong>Skipping the exit plan.</strong> The data stays open, the bucket type does not. Write down how you would move back before you move in.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is S3 Tables just Iceberg with extra steps?</h3>



<p class="wp-block-paragraph">No. It is standard Apache Iceberg with maintenance run by the service and a purpose-built bucket type underneath. The table format is identical, so engines and data stay portable. What differs is who executes maintenance and which bucket-level features you give up to get it.</p>



<h3 class="wp-block-heading">Can I get managed maintenance without moving to a table bucket?</h3>



<p class="wp-block-paragraph">Yes. AWS Glue Data Catalog table optimization brings automatic compaction, snapshot retention and orphan file management to Iceberg tables in general purpose buckets. For most teams asking this question, that is the answer they actually wanted.</p>



<h3 class="wp-block-heading">Does compaction break time travel?</h3>



<p class="wp-block-paragraph">Compaction writes a new snapshot, so history survives. Snapshot management is what removes time travel, because expiring a snapshot deletes its metadata and can delete the associated data files. If you rely on travelling to a specific point, set the minimum snapshot count and maximum snapshot age to cover that window deliberately.</p>



<h3 class="wp-block-heading">Why does my PyIceberg client get access denied on S3 Tables?</h3>



<p class="wp-block-paragraph">Usually because you are going through the Glue Data Catalog integration, where both IAM and Lake Formation must authorize the request and only the IAM half is configured. Either add the Lake Formation grants and enable external engine access, or switch to the S3 Tables Iceberg REST endpoint for direct IAM-only access.</p>



<h3 class="wp-block-heading">Is S3 Tables more expensive than regular S3?</h3>



<p class="wp-block-paragraph">The storage rate carries a premium, plus a per-object monitoring fee and compaction charges. Against that, you remove the compute you were spending on maintenance. Whether the total is higher depends almost entirely on object count, so model it from your file counts rather than your terabytes.</p>



<h3 class="wp-block-heading">Can I disable compaction and control it myself?</h3>



<p class="wp-block-paragraph">Yes, compaction can be disabled per table through the maintenance configuration API, which also stops the associated charges. Reasonable if you already have a tuned compaction job, though it undercuts most of the reason to be on a table bucket.</p>



<h3 class="wp-block-heading">Which should a small team pick?</h3>



<p class="wp-block-paragraph">A small team with one high-churn streaming table and no data platform engineer is the clearest fit for S3 Tables. A small team running a nightly batch load into a few large tables should stay on general purpose S3 and turn on Glue table optimization if maintenance is the worry.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables vs Iceberg on S3 is not really a performance comparison, and not really a cost comparison either. Both store the same Parquet under the same open table format, and either performs well when the layout is maintained.</p>



<p class="wp-block-paragraph">What you are choosing is where responsibility for compaction, snapshot expiry and orphan cleanup lives, and how confident you are that it will still be happening in two years when the person who set it up has moved on. Low confidence and a high commit rate means managed Iceberg earns its premium. A real owner, a real schedule and large batch writes means you are paying for a problem you do not have.</p>



<p class="wp-block-paragraph">Either way, check the two silent failures before you ship: table properties that disable snapshot expiry, and a writer whose compression codec compaction cannot read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or fixing a table that already rotted?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for teams running Iceberg on AWS:</p>



<ul class="wp-block-list">
<li>Auditing Iceberg tables for small file growth, snapshot bloat and orphan files, with a before-and-after on planning time and storage</li>



<li>Modelling S3 Tables against your real object counts and commit frequency, so the decision comes from your numbers rather than a pricing page example</li>



<li>Setting up AWS Glue Data Catalog table optimization as the low-risk middle path, including the IAM and network configuration it needs</li>



<li>Untangling Lake Formation and IAM when external Iceberg clients get denied through the Glue REST endpoint</li>



<li>Tuning the writer side, where the actual fix usually lives: commit batching, target file sizes, partition layout and compression choices</li>



<li>Building monitoring that makes maintenance visible, so degradation shows up on a dashboard instead of in a &#8220;why is Athena slow&#8221; ticket</li>
</ul>



<p class="wp-block-paragraph">Send me a table&#8217;s manifest summary, a Glue optimizer config, or the planning stage of a slow query, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
