<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Data Engineering | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/data-engineering/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/data-engineering/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 14 Nov 2026 07:01:31 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Data Engineering | John Nessime</title>
	<link>https://john-nessime.com/blog/data-engineering/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 14 Nov 2026 07:01:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=683</guid>

					<description><![CDATA[<p>Managed Iceberg is not a storage upgrade, it is a decision about who owns compaction, snapshot expiry and orphan cleanup. A practical comparison of S3 Tables against Iceberg in general purpose S3 buckets, covering the maintenance you actually get, the Lake Formation trap in the access path, how the two bills are shaped, and a seven-step decision procedure you can run against a single table.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The dashboard that used to load in eight seconds now takes ninety. Nobody changed the query, nobody changed the schema, and the table is barely bigger than it was last quarter. Someone opens a ticket titled &#8220;Athena slow?&#8221; and it sits there, because there is no error to point at.</p>



<p class="wp-block-paragraph">What changed is the file count. A streaming job has been writing small Parquet files every few minutes for months, and each one is another entry the query planner opens before it can plan anything. The data is fine. The layout rotted.</p>



<p class="wp-block-paragraph">That is the problem underneath the whole S3 Tables vs Iceberg on S3 argument. It is not a storage question. It is a question about who owns table maintenance, and what happens when the answer is nobody. This post covers what S3 Tables actually runs for you, where each option wins, why the access path matters more than most comparisons admit, how the two bills are shaped, and a decision procedure you can apply to a specific table.</p>



<h2 class="wp-block-heading">The failure that doesn&#8217;t announce itself</h2>



<p class="wp-block-paragraph">Iceberg on general purpose S3 works beautifully on day one. You write Parquet, commit a snapshot, Athena reads it. Nothing about that first week tells you it needs upkeep. Three things degrade quietly afterward.</p>



<ul class="wp-block-list">
<li><strong>Small files accumulate.</strong> Planning cost scales with how many files the planner considers, not with how much data you actually scan.</li>



<li><strong>Snapshots pile up.</strong> Every write creates a table version, and old snapshots keep their data files alive. Storage grows even when the row count is flat.</li>



<li><strong>Orphan files never get cleaned.</strong> A killed write can leave files no snapshot references. They are invisible to every query and they bill every month forever.</li>
</ul>



<p class="wp-block-paragraph">None of that produces an alert. No threshold crossing, no failed job, no red panel. It is a slope, not a cliff, which is why teams hit the &#8220;Athena slow?&#8221; ticket eighteen months in with nothing to blame.</p>



<p class="wp-block-paragraph">Amazon S3 Tables exists to take those three jobs off you. That is a real pitch. The question is what you hand over to accept it.</p>



<h2 class="wp-block-heading">What S3 Tables actually runs for you</h2>



<p class="wp-block-paragraph">S3 Tables introduces a distinct bucket type, the table bucket, which holds tables as subresources rather than a prefix full of objects you manage. Inside it, S3 runs three maintenance operations, all on by default. They map one to one onto the standard Apache Iceberg maintenance routines, which tells you exactly what the do-it-yourself equivalent costs in effort.</p>



<h3 class="wp-block-heading">Compaction</h3>



<p class="wp-block-paragraph">Compaction merges small objects into larger ones and applies row-level deletes while it does. It is configured per table, with a target file size defaulting to 512MB and settable as low as 64MB. It reads Parquet, Avro and ORC, and writes Parquet by default.</p>



<p class="wp-block-paragraph">Two limits matter more than the defaults. Compaction does not support the <code>Fixed</code> data type, and it does not support brotli or lz4 compression. If your writer uses lz4 because someone benchmarked it years ago, you are paying for a managed service that cannot do the job you bought it for. Check the writer config before you migrate.</p>



<p class="wp-block-paragraph">Also worth internalizing: compaction runs on an automated schedule you do not control. There is no &#8220;compact now, I have a board report in an hour&#8221; button. Between a burst of small writes and the next pass, the table sits in exactly the state you were trying to avoid.</p>



<h3 class="wp-block-heading">Snapshot management, and the trap inside it</h3>



<p class="wp-block-paragraph">Snapshot management expires old table versions. Two per-table values govern it: a minimum number of snapshots to keep, and a maximum snapshot age. Both conditions must be satisfied before anything is removed, so the minimum count wins over the age setting on a quiet table.</p>



<p class="wp-block-paragraph">Here is the part that bites. Snapshot management does not honor retention set as Iceberg table properties. Configure a branch or tag based retention policy, or a retention in the table metadata longer than what you set through the maintenance API, and snapshot management is <em>disabled</em> for that table.</p>



<p class="wp-block-paragraph">Not overridden. Disabled. S3 stops expiring snapshots, you keep paying storage for every version, and nothing tells you. Someone runs an innocuous <code>ALTER TABLE ... SET TBLPROPERTIES</code> to tag a release for reproducibility, and months later the storage line has doubled on a table nobody thought was growing. If you adopt S3 Tables, make &#8220;nobody sets snapshot retention in table properties&#8221; a written rule and audit for it.</p>



<h3 class="wp-block-heading">Unreferenced file removal</h3>



<p class="wp-block-paragraph">This is orphan cleanup. It deletes data and metadata files no longer referenced by Iceberg metadata once they pass a retention window, configured as a number of days for unreferenced files and a number of days for non-current files.</p>



<p class="wp-block-paragraph">The structural detail is scope. Compaction and snapshot management are per table. Unreferenced file removal is configured <em>at the table bucket level only</em>. You cannot give a compliance-sensitive table a one day orphan window and a scratch table ten days in the same bucket. Different retention behavior means different table buckets, and that is a design-time decision.</p>



<p class="wp-block-paragraph">Editing any of this needs the maintenance configuration permissions, and automated runs surface in CloudTrail, which is your only real visibility into whether maintenance is happening.</p>



<pre class="wp-block-code"><code>s3tables:GetTableMaintenanceConfiguration
s3tables:PutTableMaintenanceConfiguration</code></pre>



<p class="wp-block-paragraph">Grant those to whoever owns the table, not to the pipeline role. A pipeline should never be able to switch off its own cleanup.</p>



<h2 class="wp-block-heading">S3 Tables: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> A high-frequency writer with no dedicated data platform engineer. Change data capture, clickstream, IoT telemetry, log tables: anything producing a steady drip of small files that nobody will schedule a compaction job for. AWS also positions table buckets as supporting substantially higher transaction rates than general purpose buckets, which is a vendor claim worth testing against your own write pattern rather than taking from a marketing page.</p>



<p class="wp-block-paragraph">It wins again when you want maintenance to be a property of the storage rather than a job in someone&#8217;s scheduler. A compaction job in Airflow can be deleted, break silently, or lose its owner when a contract ends. Maintenance that lives in the bucket does not.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Table buckets are not general purpose buckets, and the gap is wider than the name suggests.</p>



<ul class="wp-block-list">
<li>Public access policies and presigned URLs are not supported. If any workflow hands out object URLs, that pattern is gone.</li>



<li>Table bucket and table access policies are capped at 20 KB, which sounds generous until you are writing per-tenant conditions.</li>



<li>Replication has edges: replicas are read-only and compaction does not run on them, tables with tags or branches cannot be replicated, and a table upgraded from Iceberg V2 to V3 cannot be replicated afterward.</li>



<li>Your lifecycle rules, inventory reports, bucket-level scripts and storage class habits do not transfer directly.</li>



<li>Compaction timing is not yours. Where layout freshness matters within the hour, the scheduled model is a genuine constraint.</li>
</ul>



<h2 class="wp-block-heading">Iceberg on general purpose S3: where it wins, where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give this option its real case, because the &#8220;unmanaged&#8221; framing is out of date. Iceberg tables registered in the AWS Glue Data Catalog can have automatic table optimization enabled, covering compaction, snapshot retention and orphan file management. The optimizer watches partitions and starts compaction once file counts and sizes cross a threshold. The choice is no longer managed versus manual. It is closer to managed at the storage layer versus managed at the catalog layer.</p>



<p class="wp-block-paragraph"><strong>Where it wins.</strong></p>



<ul class="wp-block-list">
<li><strong>You control timing.</strong> Compact before a heavy reporting window or after a backfill, because you call the procedure.</li>



<li><strong>One bucket type, one mental model.</strong> Lifecycle rules, storage classes, bucket policies and VPC endpoint patterns keep working.</li>



<li><strong>Cheaper base storage rate.</strong> Table bucket storage carries a premium, and at volume that is not a rounding error.</li>



<li><strong>Exit is cheap.</strong> Iceberg and Parquet in a bucket you own. Moving catalogs is a metadata operation, not a data migration.</li>



<li><strong>Mixed workloads.</strong> Raw drops, partner exports and Iceberg tables can share one storage architecture.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> You own the failure mode. If nobody enables the optimizer or schedules the procedures, nothing runs and nothing complains. The work is not complicated, which is exactly the problem: it is easy enough that everyone assumes someone else did it.</p>



<p class="wp-block-paragraph">The manual path is three Iceberg Spark procedures on a schedule you own:</p>



<pre class="wp-block-code"><code>-- consolidate small data files
CALL catalog.system.rewrite_data_files(table =&gt; 'db.events')

-- drop old table versions
CALL catalog.system.expire_snapshots(table =&gt; 'db.events')

-- delete files no snapshot references
CALL catalog.system.remove_orphan_files(table =&gt; 'db.events')</code></pre>



<p class="wp-block-paragraph">Each accepts arguments for targets and retention that you should set deliberately rather than accept by default. Treat <code>remove_orphan_files</code> with care: it compares storage against metadata, and a short retention window can delete files a concurrent write is still committing.</p>



<p class="wp-block-paragraph">The other cost is compute. Those procedures run on Spark, so an EMR cluster or a Glue job, with its own IAM, network path, failure modes and bill. That is the real thing you buy out of with S3 Tables, and it is a bigger line than people estimate.</p>



<h2 class="wp-block-heading">The access path is a bigger difference than the maintenance</h2>



<p class="wp-block-paragraph">This is the part that generates support tickets, and most comparisons skip it. There are two REST catalog paths to an S3 table, with different authorization models.</p>



<ol class="wp-block-list">
<li><strong>Direct, through the S3 Tables Iceberg REST endpoint.</strong> Any Iceberg-compatible client connects. Authorization is IAM only, via identity-based policies or resource-based policies on the table and table bucket. AWS suggests this for self-managed or partner catalogs, or basic read and write against a single table bucket.</li>



<li><strong>Through the AWS Glue Data Catalog integration.</strong> Tables register as Data Catalog objects, visible to Athena, Redshift, EMR, Glue ETL and Firehose, with third-party clients reaching them through the Glue Iceberg REST endpoint. Authorization is IAM <em>and</em> Lake Formation. Both checks must pass.</li>
</ol>



<p class="wp-block-paragraph">That second point is where teams lose an afternoon. A PyIceberg client with a correct IAM policy still gets denied, because integrating with the Data Catalog registered the tables as Lake Formation resources and no Lake Formation grant exists. The IAM policy is not wrong. It is not sufficient. For external engines to receive credentials, Lake Formation also has to permit full table access for external engines, and a data lake administrator has to apply the grants.</p>



<p class="wp-block-paragraph">If you already run Lake Formation for fine-grained access control, this is a feature and column-level permissions come along for free. If you do not, you have added a governance service, an administrator role and a second permissions model to a project that was supposed to be about file sizes. Price that in honestly.</p>



<h2 class="wp-block-heading">How the two bills are shaped</h2>



<p class="wp-block-paragraph">Skip the per-gigabyte rates. They change, and the decision does not live there. Look at the shape instead.</p>



<p class="wp-block-paragraph"><strong>S3 Tables bills on four dimensions:</strong> storage, requests, a per-object monitoring fee for every object in the table bucket, and compaction charged both per object and per byte processed. The monitoring fee is the one people miss, because it scales with object <em>count</em>. The same workload that makes compaction expensive makes monitoring expensive, and both are driven by how many small files your writer produces.</p>



<p class="wp-block-paragraph"><strong>Iceberg on general purpose S3 bills on:</strong> storage, requests, and the compute you run maintenance on. That compute is lumpy and easy to underestimate, but it is a knob you control. A weekly compaction window on spot capacity costs very differently from continuous optimization.</p>



<p class="wp-block-paragraph">One lever moves both models: <strong>batch your writes</strong>. Committing every thirty seconds instead of every five minutes multiplies object count by ten, and object count feeds the monitoring fee, the compaction fee, the planning time and the Spark hours alike. Fixing the writer is almost always cheaper than choosing storage around a bad writer.</p>



<p class="wp-block-paragraph">Whichever you pick, make the cost visible. Cost allocation tags on the table bucket, a CloudWatch or Grafana panel on object counts, and CloudTrail on maintenance events will teach you more in a month than any spreadsheet estimate teaches you today.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for S3 Tables vs Iceberg on S3</h2>



<p class="wp-block-paragraph">Work through these for a specific table, not for your platform as a whole. Different tables can legitimately land in different places.</p>



<ol class="wp-block-list">
<li><strong>Count the commits per hour.</strong> A handful of commits a day from a batch job writing large files is not a small file problem, and managed compaction solves nothing. Stay on general purpose S3.</li>



<li><strong>Check the writer&#8217;s compression and types.</strong> lz4 or brotli, or a <code>Fixed</code> column, and S3 Tables compaction will not process those files. Fix the writer or stay put.</li>



<li><strong>Ask who owns maintenance today, by name.</strong> If it is a person who will still be here next year and a job that alerts on failure, do it yourself. If the answer is a shrug, that shrug is what you are paying to replace.</li>



<li><strong>Check for presigned URLs, public policies, lifecycle rules or large access policies on this data.</strong> Any yes rules out a table bucket for that table.</li>



<li><strong>Decide the governance model before the storage model.</strong> Want Lake Formation fine-grained control? The Glue Data Catalog path is a real benefit. Want IAM-only simplicity? Use direct access and keep Lake Formation out of it.</li>



<li><strong>Confirm nothing sets snapshot retention in table properties.</strong> If your pipeline or dbt project does, change it or accept that managed snapshot expiry will be silently off.</li>



<li><strong>Try the middle option first.</strong> Enable Glue Data Catalog table optimization on the existing table. Reversible, no data movement, and it answers &#8220;is automated maintenance enough&#8221; without committing you to a new bucket type.</li>
</ol>



<p class="wp-block-paragraph">That last step is the one I reach for first in almost every case. It delivers the outcome you actually want, maintenance that runs without a human, at the cost of a checkbox rather than a migration.</p>



<h2 class="wp-block-heading">Where teams get this wrong</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a storage migration.</strong> It is an operations and governance decision that happens to involve storage.</li>



<li><strong>Migrating the wrong table.</strong> Teams pick the biggest one. The right candidate is the highest commit frequency, which is often small.</li>



<li><strong>Assuming enabled means running.</strong> Table properties can disable snapshot expiry and unsupported compression can skip compaction. Verify in CloudTrail that operations actually fire.</li>



<li><strong>Forgetting concurrency.</strong> Iceberg uses optimistic concurrency, so compaction and your writes can conflict. Compaction retries on failure; your pipeline needs retry logic too, on both sides of this comparison.</li>



<li><strong>Skipping the exit plan.</strong> The data stays open, the bucket type does not. Write down how you would move back before you move in.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is S3 Tables just Iceberg with extra steps?</h3>



<p class="wp-block-paragraph">No. It is standard Apache Iceberg with maintenance run by the service and a purpose-built bucket type underneath. The table format is identical, so engines and data stay portable. What differs is who executes maintenance and which bucket-level features you give up to get it.</p>



<h3 class="wp-block-heading">Can I get managed maintenance without moving to a table bucket?</h3>



<p class="wp-block-paragraph">Yes. AWS Glue Data Catalog table optimization brings automatic compaction, snapshot retention and orphan file management to Iceberg tables in general purpose buckets. For most teams asking this question, that is the answer they actually wanted.</p>



<h3 class="wp-block-heading">Does compaction break time travel?</h3>



<p class="wp-block-paragraph">Compaction writes a new snapshot, so history survives. Snapshot management is what removes time travel, because expiring a snapshot deletes its metadata and can delete the associated data files. If you rely on travelling to a specific point, set the minimum snapshot count and maximum snapshot age to cover that window deliberately.</p>



<h3 class="wp-block-heading">Why does my PyIceberg client get access denied on S3 Tables?</h3>



<p class="wp-block-paragraph">Usually because you are going through the Glue Data Catalog integration, where both IAM and Lake Formation must authorize the request and only the IAM half is configured. Either add the Lake Formation grants and enable external engine access, or switch to the S3 Tables Iceberg REST endpoint for direct IAM-only access.</p>



<h3 class="wp-block-heading">Is S3 Tables more expensive than regular S3?</h3>



<p class="wp-block-paragraph">The storage rate carries a premium, plus a per-object monitoring fee and compaction charges. Against that, you remove the compute you were spending on maintenance. Whether the total is higher depends almost entirely on object count, so model it from your file counts rather than your terabytes.</p>



<h3 class="wp-block-heading">Can I disable compaction and control it myself?</h3>



<p class="wp-block-paragraph">Yes, compaction can be disabled per table through the maintenance configuration API, which also stops the associated charges. Reasonable if you already have a tuned compaction job, though it undercuts most of the reason to be on a table bucket.</p>



<h3 class="wp-block-heading">Which should a small team pick?</h3>



<p class="wp-block-paragraph">A small team with one high-churn streaming table and no data platform engineer is the clearest fit for S3 Tables. A small team running a nightly batch load into a few large tables should stay on general purpose S3 and turn on Glue table optimization if maintenance is the worry.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables vs Iceberg on S3 is not really a performance comparison, and not really a cost comparison either. Both store the same Parquet under the same open table format, and either performs well when the layout is maintained.</p>



<p class="wp-block-paragraph">What you are choosing is where responsibility for compaction, snapshot expiry and orphan cleanup lives, and how confident you are that it will still be happening in two years when the person who set it up has moved on. Low confidence and a high commit rate means managed Iceberg earns its premium. A real owner, a real schedule and large batch writes means you are paying for a problem you do not have.</p>



<p class="wp-block-paragraph">Either way, check the two silent failures before you ship: table properties that disable snapshot expiry, and a writer whose compression codec compaction cannot read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or fixing a table that already rotted?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for teams running Iceberg on AWS:</p>



<ul class="wp-block-list">
<li>Auditing Iceberg tables for small file growth, snapshot bloat and orphan files, with a before-and-after on planning time and storage</li>



<li>Modelling S3 Tables against your real object counts and commit frequency, so the decision comes from your numbers rather than a pricing page example</li>



<li>Setting up AWS Glue Data Catalog table optimization as the low-risk middle path, including the IAM and network configuration it needs</li>



<li>Untangling Lake Formation and IAM when external Iceberg clients get denied through the Glue REST endpoint</li>



<li>Tuning the writer side, where the actual fix usually lives: commit batching, target file sizes, partition layout and compression choices</li>



<li>Building monitoring that makes maintenance visible, so degradation shows up on a dashboard instead of in a &#8220;why is Athena slow&#8221; ticket</li>
</ul>



<p class="wp-block-paragraph">Send me a table&#8217;s manifest summary, a Glue optimizer config, or the planning stage of a slow query, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-vs-iceberg-on-s3/">S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</title>
		<link>https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 12 Nov 2026 07:09:08 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Logistics Technology]]></category>
		<category><![CDATA[Continuous Aggregates]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Fleet Telematics]]></category>
		<category><![CDATA[Geofencing]]></category>
		<category><![CDATA[GNSS]]></category>
		<category><![CDATA[Hypertables]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Late-Arriving Data]]></category>
		<category><![CDATA[Map Matching]]></category>
		<category><![CDATA[MobilityDB]]></category>
		<category><![CDATA[PostGIS]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Time-Series Data]]></category>
		<category><![CDATA[TimescaleDB]]></category>
		<category><![CDATA[Track and Trace]]></category>
		<category><![CDATA[Trajectory Data]]></category>
		<category><![CDATA[Valhalla]]></category>
		<category><![CDATA[Vehicle Data]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=714</guid>

					<description><![CDATA[<p>Telematics devices lie in specific, documented ways: stale fixes reported as new, zero coordinates after reboot, and buffered records replayed days late. Here is how to build a fleet GPS data pipeline on PostgreSQL, PostGIS and TimescaleDB that survives all three and still answers trip and geofence queries fast.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It usually starts with a screenshot. Someone in operations sends the live map with a van sitting in the middle of a field, or off the coast entirely, and asks whether the tracker is broken. It isn&#8217;t. The tracker is doing exactly what its firmware says it should. Nobody decided what a bad point looks like before the data was written down, so now there are eleven million rows and no way to tell the real ones from the noise.</p>



<p class="wp-block-paragraph">By the time it reaches you it looks like a query problem. The trip report takes ninety seconds. The mileage total disagrees with the odometer. A geofence alert fires for a vehicle that never moved. Those are symptoms. The cause sits upstream, in the shape of the table and in the decision, made by default, to store every record the device sent as though it were true.</p>



<p class="wp-block-paragraph">This post covers how I&#8217;d build a fleet GPS data pipeline that stays queryable as it grows: what the dirty data looks like at the source, how to split storage so cleaning stays reversible, how to keep the raw table from drowning in its own volume, and why nearly every question your users ask is a question about trips rather than points.</p>



<h2 class="wp-block-heading">The four ways GPS data goes ugly before you query it</h2>



<p class="wp-block-paragraph">Organise your defences by failure family, not by field. Each of these produces rows that are individually well-formed and collectively wrong.</p>



<h3 class="wp-block-heading">The device sends its last known position and calls it new</h3>



<p class="wp-block-paragraph">This is the one that bites hardest, because it is invisible. When a unit loses its fix, plenty of firmware does not send a null. Teltonika&#8217;s FMB documentation is explicit: when a record has no valid coordinates because there was no fix at acquisition time, the longitude, latitude and altitude fields carry the <em>last valid fix</em>, while angle, satellites and speed are set to zero. The row looks normal. It has a timestamp, it has coordinates, it parses.</p>



<p class="wp-block-paragraph">That is worse than a gap. A van parked underground for six hours emits hundreds of records claiming the same coordinates, and your &#8220;last seen&#8221; logic reports it sitting at the entrance the whole time. Then it drives out, gets a fix elsewhere, and the trip builder draws a straight line through everything in between.</p>



<p class="wp-block-paragraph">The signal you want is the satellite count and the dilution of precision, not the coordinates. Store both. Teltonika exposes a GPS Accuracy setting expressed as a minimum HDOP, PDOP or VDOP before the device refreshes its GPS data, plus a minimum satellite count, so the hardware can be told to be pickier before it transmits. Configure that, and still keep the fields, because you will want to re-derive the threshold later without re-collecting the fleet.</p>



<h3 class="wp-block-heading">Null Island, and the reboot that gets you there</h3>



<p class="wp-block-paragraph">Zero latitude and zero longitude is a real place off the coast of Ghana, and it is where a surprising share of any fleet appears to live. The same Teltonika documentation notes that after a reboot the device sends zero coordinates. A naive pipeline stores that as a point, and every map, bounding box and distance calculation now carries an outlier several thousand kilometres from the fleet. Trivially filterable; it survives into production because nobody looks until the map is built, and by then the filter has to be retrofitted into six queries.</p>



<h3 class="wp-block-heading">Records that arrive days late, in the wrong order</h3>



<p class="wp-block-paragraph">Telematics units buffer. A vehicle drives through a tunnel or a rural dead zone, keeps recording, and dumps the backlog on reconnect. Devices in this class also let you choose whether records are saved after a position fix, after a time sync, or always, which means you can receive records whose device clock was never synchronised at all.</p>



<p class="wp-block-paragraph">So the timestamp on the record and the moment you received it are two different facts, and you need both. Storing one is the most common design mistake I see in fleet ingest. Keep only the device timestamp and you cannot answer &#8220;what did the dashboard show at 4pm yesterday&#8221;, which is what every dispute becomes. Keep only the receive time and your trips are wrong.</p>



<h3 class="wp-block-heading">Parked-vehicle drift</h3>



<p class="wp-block-paragraph">A stationary receiver with a marginal sky view reports a slow random walk of a few metres at a time, all night, in a depot. Sum the distance between consecutive points and you have invented a three-kilometre trip. Multiply by a fleet and the mileage report is one nobody trusts.</p>



<p class="wp-block-paragraph">Drift is not caught by the same rule as teleporting. A jump filter catches physically impossible movement. Drift is physically possible, it just isn&#8217;t real. You need ignition state, reported speed or an accelerometer movement flag to tell &#8220;not moving&#8221; from &#8220;moving very slowly in traffic&#8221;.</p>



<h2 class="wp-block-heading">Split the fleet GPS data pipeline into three tables</h2>



<p class="wp-block-paragraph">The instinct is one positions table with cleaning applied on write. Resist it. Once you discard a record you cannot revisit the decision, and you will change your mind about the thresholds at least twice.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Everything the device sent, as sent, plus a receive timestamp. Append-only, never cleaned in place.</li>



<li><strong>Clean.</strong> Points that passed your quality rules, with a geometry column. Derived from raw and fully rebuildable from it.</li>



<li><strong>Trips.</strong> One row per journey, with the whole path as a single geometry. This is what nearly every user-facing query reads.</li>
</ol>



<p class="wp-block-paragraph">Raw is what makes the rest safe. When a mileage figure is disputed eight months from now, you can re-run the cleaning under the old rules and the new ones and show the difference. Two timestamps, quality fields as first-class columns, and anything device-specific in JSONB so new hardware doesn&#8217;t force a migration:</p>



<pre class="wp-block-code"><code>CREATE TABLE gps_raw (
    device_id   text        NOT NULL,
    recorded_at timestamptz NOT NULL,   -- the device's clock
    received_at timestamptz NOT NULL DEFAULT now(),
    lon         double precision,
    lat         double precision,
    speed_kph   double precision,
    heading     double precision,
    satellites  smallint,
    hdop        double precision,
    ignition    boolean,
    payload     jsonb       NOT NULL DEFAULT '{}'::jsonb
);</code></pre>



<p class="wp-block-paragraph">No primary key on device and time, deliberately. Devices resend. A unique constraint here turns a harmless duplicate into an ingest failure at three in the morning. Deduplicate on the way into the clean table, where a conflict is a no-op rather than an incident.</p>



<h2 class="wp-block-heading">Making the raw table survive its own growth</h2>



<p class="wp-block-paragraph">A few hundred vehicles reporting every few seconds produces a table that ordinary PostgreSQL handles for a while and then doesn&#8217;t, usually right after someone adds a date filter to a dashboard. Time partitioning is the fix, either through native declarative partitioning or the TimescaleDB extension.</p>



<pre class="wp-block-code"><code>SELECT create_hypertable('gps_raw', 'recorded_at',
                         chunk_time_interval =&gt; INTERVAL '1 day');</code></pre>



<p class="wp-block-paragraph">Pick the chunk interval from your query pattern, not your ingest rate. If dashboards ask for &#8220;today&#8221; and &#8220;this week&#8221;, daily chunks let the planner exclude almost everything. If reporting is monthly, daily chunks leave a lot of small chunks to plan across.</p>



<p class="wp-block-paragraph">Compression is where the storage argument is won, and the segment key is what people get wrong. Segmenting by vehicle groups each vehicle&#8217;s rows together inside a compressed batch, which is what keeps &#8220;one vehicle, one day&#8221; fast afterwards:</p>



<pre class="wp-block-code"><code>ALTER TABLE gps_raw SET (
    timescaledb.compress,
    timescaledb.compress_orderby   = 'recorded_at',
    timescaledb.compress_segmentby = 'device_id'
);</code></pre>



<h3 class="wp-block-heading">The licensing detail that catches you at runtime</h3>



<p class="wp-block-paragraph">TimescaleDB ships in two editions. The Apache 2.0 build gives you hypertables. Compression, continuous aggregates and retention policies live in the Community edition under the Timescale License, which is source-available rather than OSI-open-source. The vendor states the practical boundary plainly: as long as you are not offering TimescaleDB itself as a hosted database service, you can run the Community features free and self-hosted in production.</p>



<p class="wp-block-paragraph">The trap is that this fails at runtime, not at install. If your distribution or provider ships the Apache-licensed module, <code>CREATE EXTENSION</code> succeeds, the hypertable works, and then the <code>ALTER TABLE</code> above returns an error saying the functionality is not supported under the current licence. Zabbix users have filed bugs about exactly this. Check the edition before you design around compression, especially on managed PostgreSQL where you don&#8217;t choose the build.</p>



<p class="wp-block-paragraph">If that blocks you, native partitioning plus a scheduled job rolling old partitions into archive storage is a fine answer. More code, less magic, no licence question. The related fork is whether to self-host at all: a telemetry database is steady, disk-heavy and predictable, which is where a large VPS from Hetzner, Contabo or InterServer compares well against a managed hourly bill, and also where you inherit backups, upgrades and the extension build. My <a href="https://john-nessime.com/tools/vps-vs-managed-aws/">VPS versus managed AWS calculator</a> prices your own admin hours against the managed bill and returns a break-even rather than an opinion.</p>



<h3 class="wp-block-heading">Rollups, and the late records that quietly miss them</h3>



<p class="wp-block-paragraph">Continuous aggregates pre-compute the hourly numbers so dashboards stop scanning points:</p>



<pre class="wp-block-code"><code>CREATE MATERIALIZED VIEW vehicle_hourly
WITH (timescaledb.continuous) AS
SELECT device_id,
       time_bucket(INTERVAL '1 hour', recorded_at) AS bucket,
       count(*)        AS fixes,
       max(speed_kph)  AS max_speed_kph
FROM gps_clean
GROUP BY device_id, bucket;</code></pre>



<p class="wp-block-paragraph">Now connect that to buffering. The refresh policy has a window, and anything older than its start is never revisited:</p>



<pre class="wp-block-code"><code>SELECT add_continuous_aggregate_policy('vehicle_hourly',
    start_offset      =&gt; INTERVAL '7 days',
    end_offset        =&gt; INTERVAL '1 hour',
    schedule_interval =&gt; INTERVAL '1 hour');</code></pre>



<p class="wp-block-paragraph">A device out of coverage for ten days reconnects, replays its backlog, the rows land in the clean table, and the hourly rollup for those ten days stays exactly as wrong as it was. Raw is right, the dashboard is wrong, nothing errors. Set the start offset wider than your worst realistic backlog, and graph the gap between <code>recorded_at</code> and <code>received_at</code> per device model so you know what that worst case actually is. If Grafana already points at this database, it costs you one panel.</p>



<h2 class="wp-block-heading">Cleaning: two filters that do most of the work</h2>



<p class="wp-block-paragraph">Promotion from raw to clean is where the quality rules live. Make it an idempotent batch over a time window so any period can be re-run without duplicating anything.</p>



<pre class="wp-block-code"><code>INSERT INTO gps_clean (device_id, recorded_at, geom, speed_kph)
SELECT device_id,
       recorded_at,
       ST_SetSRID(ST_MakePoint(lon, lat), 4326),
       speed_kph
FROM gps_raw
WHERE recorded_at &gt;= $1
  AND recorded_at &lt;  $2
  AND lat IS NOT NULL
  AND lon IS NOT NULL
  AND NOT (lat = 0 AND lon = 0)
  AND satellites &gt;= 4
  AND hdop &lt;= 5
ON CONFLICT (device_id, recorded_at) DO NOTHING;</code></pre>



<p class="wp-block-paragraph">Treat those numbers as starting points to tune against your own hardware, not constants. Four satellites is the minimum for a three-dimensional fix; the dilution threshold depends on your receivers and your operating environment. Write the chosen values somewhere a human will find them, because in a year someone will ask why a specific point was dropped.</p>



<p class="wp-block-paragraph">The second filter is the jump test, which needs the previous row. Cast to <code>geography</code> so distance comes back in metres rather than degrees:</p>



<pre class="wp-block-code"><code>WITH stepped AS (
  SELECT device_id, recorded_at, geom,
         LAG(geom)        OVER w AS prev_geom,
         LAG(recorded_at) OVER w AS prev_at
  FROM gps_clean
  WHERE device_id = $1
    AND recorded_at &gt;= $2
  WINDOW w AS (PARTITION BY device_id ORDER BY recorded_at)
)
SELECT device_id,
       recorded_at,
       ST_Distance(geom::geography, prev_geom::geography) AS metres,
       EXTRACT(epoch FROM recorded_at - prev_at)          AS seconds
FROM stepped
WHERE prev_geom IS NOT NULL
  AND ST_Distance(geom::geography, prev_geom::geography)
      / NULLIF(EXTRACT(epoch FROM recorded_at - prev_at), 0) &gt; 60;</code></pre>



<p class="wp-block-paragraph">Sixty metres per second is a little over 200 km/h. No delivery van does that, so anything above it is a bad fix, a clock problem, or two vehicles sharing a device identifier. Flag them rather than deleting them; the pattern tells you which of the three you have.</p>



<h2 class="wp-block-heading">Trips are the abstraction that makes queries fast</h2>



<p class="wp-block-paragraph">Users don&#8217;t ask point questions. They ask trip questions: how far did this vehicle go, when did it enter the site, how long was it stopped, which vehicles passed through this area. Answering those from a point table means aggregating thousands of rows per answer, every time.</p>



<p class="wp-block-paragraph">PostGIS has a native representation for this. A trajectory is a LINESTRING whose M dimension carries time, with M increasing from each vertex to the next, and <code>ST_IsValidTrajectory</code> checks exactly that. Once your paths are valid trajectories the spatio-temporal functions come free: <code>ST_ClosestPointOfApproach</code> returns the measure at which two moving objects were nearest, <code>ST_DistanceCPA</code> returns how near, and <code>ST_CPAWithin</code> gives the boolean. That turns &#8220;were these two vehicles ever within fifty metres of each other&#8221; into one function call instead of a self-join across a point table.</p>



<pre class="wp-block-code"><code>CREATE TABLE trips (
    trip_id     bigserial PRIMARY KEY,
    device_id   text        NOT NULL,
    started_at  timestamptz NOT NULL,
    ended_at    timestamptz NOT NULL,
    path        geometry(LineStringM, 4326) NOT NULL,
    distance_m  double precision NOT NULL,
    point_count integer     NOT NULL
);

CREATE INDEX trips_path_gix ON trips USING GIST (path);
CREATE INDEX trips_device_time_idx ON trips (device_id, started_at DESC);</code></pre>



<p class="wp-block-paragraph">Building the path is an aggregate over the clean points, with epoch seconds as the M value:</p>



<pre class="wp-block-code"><code>SELECT ST_MakeLine(
         ST_MakePointM(ST_X(geom), ST_Y(geom),
                       EXTRACT(epoch FROM recorded_at))
         ORDER BY recorded_at
       ) AS path
FROM gps_clean
WHERE device_id = $1
  AND recorded_at &gt;= $2
  AND recorded_at &lt;  $3;</code></pre>



<p class="wp-block-paragraph">Segmenting points into trips is a business decision dressed as a technical one. Ignition on to ignition off is cleanest where the wiring supports it; otherwise speed below a floor for longer than a gap threshold ends the trip. Whichever you pick, store the rule as a column on the trip, because someone will change it and you&#8217;ll need to know which rows were built under which version.</p>



<p class="wp-block-paragraph">For rendering, keep a simplified two-dimensional copy. <code>ST_SimplifyPreserveTopology</code> runs Douglas-Peucker without collapsing the line into nonsense, and a route drawn at city zoom doesn&#8217;t need a vertex every two seconds. Simplify for display only, never for the geometry you measure distance from.</p>



<h2 class="wp-block-heading">Map matching: when to snap, and when to skip it</h2>



<p class="wp-block-paragraph">Snapping traces to the road network fixes a cosmetic problem, paths wandering across the pavement, and a real one, distance summed from raw points overstating the journey. Valhalla&#8217;s Meili implements the well-known hidden Markov approach to matching noisy, sparse traces. Its API exposes two actions: <code>trace_route</code> returns the shape snapped to the road network with directions, and <code>trace_attributes</code> returns per-segment attribution along the matched path. Requests should be POSTed, because an encoded trace gets large. OSRM&#8217;s match service solves the same problem with a similar model.</p>



<pre class="wp-block-code"><code>curl -s -X POST https://valhalla.internal.example/trace_attributes 
  -H 'Content-Type: application/json' 
  -d '{
        "shape": [
          {"lat": 52.0907, "lon": 5.1214},
          {"lat": 52.0912, "lon": 5.1229}
        ],
        "costing": "auto",
        "shape_match": "map_snap"
      }'</code></pre>



<p class="wp-block-paragraph">Two things decide whether it&#8217;s worth running. First, trace density: Valhalla&#8217;s own guidance is to keep points roughly between one per second and one per ten seconds, and warns the match degrades as that spacing widens. If your devices report every two minutes to save data, matching will confidently snap you to plausible roads you never used, which is worse than an obviously wobbly line because it looks correct.</p>



<p class="wp-block-paragraph">Second, what the number is for. Billing, driver disputes and compliance want the matched path. An internal ops map does not. Matching is either a per-trip network call or a self-hosted routing service with a tile build to maintain, and it&#8217;s the most operationally expensive piece in this design. If no money rides on the number, skip it and say so.</p>



<p class="wp-block-paragraph">Self-hosting, Valhalla and OSRM both run fine in containers against an OpenStreetMap extract, on a separate box from the one serving queries because the tile build is memory-hungry. Worth pasting that stack into my <a href="https://john-nessime.com/tools/docker-compose-production-checker/">Docker Compose production checker</a> first, since routing containers are exactly the kind of thing that quietly exposes a port or writes tiles to a volume that dies on restart.</p>



<h2 class="wp-block-heading">Geofences that actually use the index</h2>



<p class="wp-block-paragraph">Geofence queries are where a working platform turns slow, and the cause is nearly always a function that can&#8217;t use the spatial index. <code>ST_Intersects</code>, <code>ST_Within</code> and <code>ST_DWithin</code> are index-assisted. <code>ST_Distance</code> is not. Writing <code>WHERE ST_Distance(a, b) &lt; 500</code> gets the right answer by computing distance for every candidate row.</p>



<pre class="wp-block-code"><code>-- Polygon fences: which trips crossed a depot this week
SELECT g.name, t.trip_id, t.device_id
FROM trips t
JOIN geofences g ON ST_Intersects(t.path, g.area)
WHERE t.started_at &gt;= now() - INTERVAL '7 days';

-- Radius fences: within 500 m of a point, using the index
SELECT t.trip_id
FROM trips t
WHERE ST_DWithin(t.path::geography, $1::geography, 500);</code></pre>



<p class="wp-block-paragraph">Running fences against trips rather than points is the other half of the win. A hundred fences against ten million points is a hundred million comparisons. A hundred fences against fifty thousand trip geometries, each with a bounding box the index rejects in one step, is a different query entirely.</p>



<p class="wp-block-paragraph">The catch is latency. Trips only exist once they end, so live entry and exit alerts still run against incoming points. Keep that path separate and deliberately small: active fences only, evaluated as records arrive. Don&#8217;t make one query serve both the live alert and the monthly report.</p>



<h2 class="wp-block-heading">Troubleshooting a fleet data platform that has gone wrong</h2>



<p class="wp-block-paragraph"><strong>Mileage disagrees with the odometer.</strong> Compare summed point-to-point distance against the matched-path distance for the same trip. Consistently higher means drift or noise, so tighten the quality filter and check whether stationary periods are included. Lower means missing points, with gaps bridged by straight lines.</p>



<p class="wp-block-paragraph"><strong>A dashboard number is wrong but the raw data is right.</strong> Suspect the rollup window first. Query the raw table and the continuous aggregate for the same bucket and compare. A difference means late data outside the refresh offset.</p>



<p class="wp-block-paragraph"><strong>A vehicle sits still for hours, then teleports.</strong> Check satellites and HDOP across the stationary block. Zero satellites throughout means repeated stale fixes, not a parked vehicle.</p>



<p class="wp-block-paragraph"><strong>Queries slowed suddenly with no traffic change.</strong> Check whether the planner is excluding chunks. A predicate wrapped in a function on the time column, or a parameter type that doesn&#8217;t match, prevents partition pruning and silently turns a one-day scan into a full-table scan. Run <code>EXPLAIN</code> and count the chunks in the plan.</p>



<p class="wp-block-paragraph"><strong>Two vehicles occupy the same position.</strong> Almost always a device moved between vehicles without the assignment history being updated. Model the device-to-vehicle link as a table with validity intervals, not a column on the vehicle.</p>



<p class="wp-block-paragraph">One failure worth rehearsing before it happens is the ingest endpoint going down. Devices buffer, so a short outage survives and a long one becomes a thundering replay on recovery. If you&#8217;ve never asked what breaks when the database node dies mid-shift, my <a href="https://john-nessime.com/tools/cloud-architecture-assessment/">cloud architecture assessment</a> scores where the single points of failure sit.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Cleaning on ingest and keeping nothing else. The rules will change and you&#8217;ll have no way to apply them retroactively.</li>



<li>Storing latitude and longitude as plain numeric columns. Fine until the first spatial query, then you&#8217;re building indexes on a table you can&#8217;t afford to lock.</li>



<li>Treating the device timestamp as authoritative. Unsynchronised clocks and buffered replays make it a claim, not a fact.</li>



<li>Using <code>ST_Distance</code> in a WHERE clause instead of <code>ST_DWithin</code>. Same answer, no index.</li>



<li>Designing around compression or continuous aggregates without checking which TimescaleDB edition is installed.</li>



<li>Map matching a sparse trace and trusting the output. Confident wrong answers are harder to catch than obvious ones.</li>



<li>Putting live alerting and historical reporting through the same queries.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw append-only and every downstream table rebuildable from it. Test that rebuild before you need it.</li>



<li>Store both timestamps on every record, and alert on the lag between them per device model.</li>



<li>Keep the quality fields after filtering on them. Satellites and HDOP are how you re-derive a threshold later.</li>



<li>Match the chunk interval to how people query, and revisit it when the reporting cadence changes.</li>



<li>Answer user questions from trips. Reach for points only when you genuinely need sub-trip resolution.</li>



<li>Record the segmentation rule and quality thresholds as data, not as code comments.</li>



<li>Retire raw data on a written retention schedule rather than when the disk fills.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need TimescaleDB, or is plain PostgreSQL enough?</h3>



<p class="wp-block-paragraph">Plain PostgreSQL with native partitioning and PostGIS carries a small fleet a long way, with no licensing question attached. TimescaleDB earns its place when you want compression and continuous aggregates without maintaining that machinery yourself. It saves you code, not capability.</p>



<h3 class="wp-block-heading">Should I use PostGIS geometry or geography for GPS points?</h3>



<p class="wp-block-paragraph">Store as <code>geometry</code> in SRID 4326 and cast to <code>geography</code> for distance and radius work. Geometry has broader function coverage and cheaper operations; geography gives metres on a sphere instead of degrees. Casting at the point of use is the pragmatic middle.</p>



<h3 class="wp-block-heading">How long should I keep raw GPS points?</h3>



<p class="wp-block-paragraph">Long enough to cover the dispute window your business actually faces, which is driven by employment law, insurance or customer contracts rather than by engineering. Decide it with whoever owns that risk, write it down, enforce it with a retention policy. Trips are far smaller and usually worth keeping much longer.</p>



<h3 class="wp-block-heading">What about ClickHouse or InfluxDB instead?</h3>



<p class="wp-block-paragraph">Both are genuinely strong at high-volume ingest and time-range scans, and for a purely time-series workload I&#8217;d take either seriously. The reason I reach for PostgreSQL here is that fleet data isn&#8217;t purely time-series: it joins constantly to vehicles, drivers, contracts and sites, and it needs spatial predicates in the same query as the time filter. One database beats two and a reconciliation job. If your spatial needs are trivial and your volume is enormous, that flips.</p>



<h3 class="wp-block-heading">Is MobilityDB worth adding?</h3>



<p class="wp-block-paragraph">MobilityDB extends PostgreSQL and PostGIS with proper moving-object types, <code>tgeompoint</code> and <code>tgeogpoint</code>, that interpolate between observations so you can query a position at any instant. It&#8217;s an OSGeo project and the modelling is better than hand-rolled trajectories. The cost is another extension to install and upgrade, including <code>shared_preload_libraries</code> and locking configuration on the server. I&#8217;d add it when trajectory analysis is the product, not when it&#8217;s a reporting feature.</p>



<h3 class="wp-block-heading">How do I stop parked vehicles generating fake mileage?</h3>



<p class="wp-block-paragraph">Don&#8217;t rely on distance alone, because drift is physically plausible. Use ignition state or a movement flag to mark stationary periods, exclude those points from distance summation, and keep them in raw so the stop duration stays auditable.</p>



<h3 class="wp-block-heading">Where does a fleet GPS data pipeline break first at scale?</h3>



<p class="wp-block-paragraph">The dashboard queries, and nearly always for the same reason: something in the predicate stopped the planner pruning partitions. Before adding hardware, run <code>EXPLAIN</code> and confirm how many chunks the query touches. Adding vehicles rarely breaks a partitioned table. Bad predicates always do.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A fleet GPS data pipeline stays queryable because of two decisions made early, not because of the database you picked. Keep the raw record untouched so every cleaning rule stays reversible, and answer questions from trips rather than points so the expensive aggregation happens once instead of on every page load.</p>



<p class="wp-block-paragraph">Everything else is tuning. Chunk intervals, compression settings and index choices all matter, and all of them are recoverable. Throwing away the record of what the device actually sent is the only mistake here you can&#8217;t undo later.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">Most fleet data work arrives as a number somebody disputes, and the fix lives three layers upstream of the report. Things I&#8217;d take on here:</p>



<ul class="wp-block-list">
<li>Reviewing an existing positions schema and working out what it takes to make it partition-friendly without a full rewrite.</li>



<li>Designing the raw, clean and trip split, including the idempotent promotion job so any window can be safely re-run.</li>



<li>Tracking down why a rollup and the raw data disagree, which is usually late records falling outside a refresh window.</li>



<li>Building quality filters against your actual hardware, so thresholds fit your receivers rather than someone else&#8217;s blog post.</li>



<li>Getting geofence and trip queries onto the spatial index, with the EXPLAIN output to show what changed.</li>



<li>Standing up self-hosted map matching, and deciding honestly if you need it at all.</li>
</ul>



<p class="wp-block-paragraph">If any of that is live for you, send the concrete thing: a table definition, the <code>EXPLAIN</code> output from the slow query, or a day of records from one device. That&#8217;s usually enough to say something useful straight away.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/data-pipeline-review/">Data pipeline rerun review</a></div>



<div class="wp-block-button is-style-outline"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Athena Partitioning: Why Your Queries Are Still Slow</title>
		<link>https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 07:01:44 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CTAS]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Predicate Pushdown]]></category>
		<category><![CDATA[Query Performance]]></category>
		<category><![CDATA[Small Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=399</guid>

					<description><![CDATA[<p>Partitioning an Athena table is not the same as getting the benefit of it. A walkthrough of the three failure families behind slow queries on partitioned external data: predicates that never prune, catalog lookups that dominate planning, and file layouts that undo everything. Includes partition projection trade-offs, Glue partition indexes, compaction patterns, and how to tell which problem you have.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/">Athena Partitioning: Why Your Queries Are Still Slow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the daily rollup was timing out. Same query, same workgroup, same table, nobody had touched the SQL in months. The only thing that had changed was that the table kept collecting data.</p>



<p class="wp-block-paragraph">That is the part that confuses people, because the table <em>was</em> partitioned. Someone had done the work. The S3 prefixes had tidy <code>key=value</code> folders, the Glue catalog knew about them, and the query had a <code>WHERE</code> clause on the date. By every checklist on the internet, this table was fine.</p>



<p class="wp-block-paragraph">Athena partitioning fails quietly in a way that other performance problems do not. A query that reads every byte of a five terabyte table does not throw an error. It returns the correct answer. It just takes four minutes instead of eight seconds and bills you for the privilege, and the only place the truth shows up is in the bytes-scanned number that nobody looks at until the finance channel does.</p>



<p class="wp-block-paragraph">This post is about the gap between having partitions and getting the benefit of them. I will go through the three failure families that actually cause slow Athena queries on partitioned external data, how to tell which one you are in, and what to do about each. There is some setup DDL along the way, but the point is the mechanism, not the syntax.</p>



<h2 class="wp-block-heading">How Athena decides what to read</h2>



<p class="wp-block-paragraph">Before any of the failures make sense, it helps to be precise about what happens between pressing run and bytes moving off S3.</p>



<p class="wp-block-paragraph">Athena does not scan a table. It builds a list of S3 objects and reads those. Getting to that list involves three separate reductions, and each one can fail independently of the others:</p>



<ol class="wp-block-list">
<li><strong>Partition resolution.</strong> Athena asks the AWS Glue Data Catalog for the partitions belonging to the table. For a Hive-style table this is a <code>GetPartitions</code> call, and it happens before any filtering of your data.</li>

<li><strong>Partition pruning.</strong> Athena compares your <code>WHERE</code> clause against those partition values and throws away the prefixes that cannot match. Objects under discarded prefixes are never fetched at all.</li>

<li><strong>In-file skipping.</strong> Within the surviving files, columnar formats let Athena read only the columns you asked for, and use row group statistics to skip chunks whose min and max values rule them out.</li>
</ol>



<p class="wp-block-paragraph">Step two is the one people mean when they say partitioning. Step one is the one that quietly becomes the bottleneck on tables with a lot of partitions. Step three is the one that gets wiped out by bad file layout even when the first two worked perfectly.</p>



<p class="wp-block-paragraph">Each reduction only helps if the one above it fired. That is the whole shape of the problem.</p>



<h2 class="wp-block-heading">Failure family one: the filter that never pruned</h2>



<p class="wp-block-paragraph">This is the most common one and the least visible. The table is partitioned correctly. The query filters on the partition column. Athena still reads everything.</p>



<p class="wp-block-paragraph">Pruning happens during planning, before Athena has seen a single row. So the planner has to be able to evaluate your predicate against literal partition values using nothing but the query text. Anything that defers the value until runtime defeats it.</p>



<p class="wp-block-paragraph">The usual culprits:</p>



<ul class="wp-block-list">
<li><strong>A function wrapped around the partition column.</strong> Partition values live in the prefix as text. If you write something that parses or casts them before comparing, the planner is comparing the output of a function it has not run yet, so it keeps every partition.</li>

<li><strong>A filter that arrives through a join.</strong> If the partition value comes from the other side of a join rather than from a literal, planning has no value to prune with. This is the expensive one, because the query looks careful. Joining a small control table that says which day to load is a very natural thing to write and a very reliable way to scan the whole fact table.</li>

<li><strong>A subquery that produces the value.</strong> Same problem, same reason.</li>

<li><strong>Filtering the wrong column.</strong> Tables often carry both an event timestamp inside the data and a partition column derived from it. Filtering the timestamp is correct and returns the right rows. It also prunes nothing.</li>
</ul>



<p class="wp-block-paragraph">You do not have to guess. Run the query through <code>EXPLAIN</code> and look at what the table scan node says it is doing.</p>



<pre class="wp-block-code"><code>EXPLAIN
SELECT status, count(*)
FROM logs.access_events
WHERE dt = '2024-05-01';</code></pre>



<p class="wp-block-paragraph">The plan shows the constraint applied to the source. If your partition column appears there with a bound value, pruning will happen. If the predicate has been pushed into a filter node above the scan instead, it will not, and you are about to read the table.</p>



<p class="wp-block-paragraph">The cheaper check is behavioral: run the query with a narrow filter and note the bytes scanned, then widen the filter by ten times and run it again. If the number barely moves, nothing is pruning. Two runs and you have your answer without reading a plan.</p>



<p class="wp-block-paragraph">The fix for the join and subquery cases is unglamorous: resolve the value first, then substitute it into the query as a literal. In an orchestrated pipeline that means one step that computes the partition to load and a second step that templates it into the SQL. Step Functions, Airflow, or a shell script all do this fine. It feels like a step backwards from a single elegant query. It is the difference between reading one prefix and reading all of them.</p>



<h2 class="wp-block-heading">Failure family two: the partition list is the bottleneck</h2>



<p class="wp-block-paragraph">Now the opposite situation. Pruning works, the query reads a tiny slice, and the thing is still slow. Watch where the time goes and you will find most of it elapses before any data is read.</p>



<p class="wp-block-paragraph">That is partition resolution. Athena has to retrieve the partition metadata from Glue before it can prune it, and on a table with hundreds of thousands of partitions that retrieval is a real remote operation with real latency. AWS documents that Athena can query Glue tables with up to ten million partitions but cannot read more than one million partitions in a single scan, which is a ceiling worth knowing about before you design a scheme that approaches it.</p>



<p class="wp-block-paragraph">You get a strong hint from a query that scans almost nothing and still takes twenty seconds. Planning time is not proportional to data volume. It is proportional to partition count.</p>



<h3 class="wp-block-heading">Option one: Glue partition indexes</h3>



<p class="wp-block-paragraph">Glue lets you create an index on a combination of partition keys, which lets the catalog filter server-side rather than handing Athena the full list. Creating the index alone does nothing for Athena. You also have to flip a table property, and this is the step people miss:</p>



<pre class="wp-block-code"><code>ALTER TABLE logs.access_events
SET TBLPROPERTIES ('partition_filtering.enabled' = 'true');</code></pre>



<p class="wp-block-paragraph">The index makes filtering possible on the Glue side; the property tells Athena to actually use it. Set the same property to <code>false</code> to back the change out without dropping the index, which makes this an easy thing to try and easy to reverse.</p>



<p class="wp-block-paragraph">This is the option I reach for first when the table already exists, already has registered partitions, and is also read by other engines. It changes nothing about how the table works.</p>



<h3 class="wp-block-heading">Option two: partition projection</h3>



<p class="wp-block-paragraph">Partition projection takes a different approach. Instead of asking Glue what partitions exist, you describe the pattern in table properties and Athena computes the partition values and their S3 locations in memory. The <code>GetPartitions</code> call disappears entirely.</p>



<p class="wp-block-paragraph">It also solves a maintenance problem. There is nothing to register when new data lands, so no crawler schedule and no repair command. For time-series data with a predictable layout, this is usually the right answer.</p>



<p class="wp-block-paragraph">A date-partitioned table looks like this:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE logs.access_events (
  request_id string,
  status int,
  path string,
  event_time timestamp
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://my-lake/access-events/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.dt.type' = 'date',
  'projection.dt.format' = 'yyyy-MM-dd',
  'projection.dt.range' = '2021-01-01,NOW',
  'projection.dt.interval' = '1',
  'projection.dt.interval.unit' = 'DAYS'
);</code></pre>



<p class="wp-block-paragraph">The <code>range</code> property accepts relative expressions such as <code>NOW</code>, <code>NOW-3YEARS</code> and <code>NOW+1MONTH</code>, which is what makes the table self-maintaining. The <code>format</code> string follows Java&#8217;s date formatter, and the same format must be used for the range bounds.</p>



<p class="wp-block-paragraph">If your prefixes do not follow the standard <code>column=value</code> layout, which is common for Firehose delivery streams and service logs, add a template that maps partition values onto your actual paths:</p>



<pre class="wp-block-code"><code>'storage.location.template' = 's3://my-lake/access-events/${dt}/'</code></pre>



<p class="wp-block-paragraph">Every partition column must appear in the template, and the path must end in a slash.</p>



<h3 class="wp-block-heading">Where partition projection hurts</h3>



<p class="wp-block-paragraph">Projection is the most oversold feature in this whole area, so here are the edges honestly.</p>



<ul class="wp-block-list">
<li><strong>Out-of-range queries return zero rows, not an error.</strong> Query a date before your configured range starts and Athena completes successfully with an empty result. This is the single nastiest behavior in the feature, because a dashboard that silently goes blank looks like a data problem rather than a configuration problem.</li>

<li><strong>Empty partitions cost you.</strong> Athena projects partitions whether or not they exist in S3. AWS&#8217;s own guidance is that if more than half your projected partitions are empty, you are better off with traditional registered partitions.</li>

<li><strong>Enabling it makes Athena ignore registered partition metadata for that table.</strong> Any hand-registered partition pointing somewhere unusual stops being honored.</li>

<li><strong>It is Athena-only.</strong> Redshift Spectrum, EMR and Athena for Spark reading the same table fall back to standard partition metadata. If your lake is shared, projection fixes one consumer and leaves the rest as they were.</li>

<li><strong><code>SHOW PARTITIONS</code> stops being useful.</strong> Projected partitions are not in the catalog, so the command does not list them. Tooling that introspects partitions will need another approach.</li>

<li><strong>Views do not inherit it.</strong> Projection is configured on tables. Set it on the underlying tables, not on views over them.</li>

<li><strong>Projected dates are generated in UTC.</strong> If your prefixes are written in local time, the boundaries will not line up and you will lose or duplicate a day at the edges.</li>
</ul>



<p class="wp-block-paragraph">The <code>enum</code> type deserves its own warning. It is tempting for things like customer IDs or account codes, but AWS recommends keeping enum projections to a few dozen values, because the whole projection configuration lives inside the Glue table metadata and that metadata has a size ceiling. If you need high-cardinality identifiers as a partition key, the <code>injected</code> type is the right tool, with the caveat that queries fail unless every injected column has a filter, and the <code>IN</code> list is capped at a thousand values.</p>



<h2 class="wp-block-heading">Failure family three: too many partitions and too many files</h2>



<p class="wp-block-paragraph">This is the one people create for themselves while trying to fix the first two. Partitioning felt good, so they partitioned harder. Year, month, day, hour, region, tenant. The prefixes look magnificent. The queries get worse.</p>



<p class="wp-block-paragraph">Two mechanisms are working against you at once.</p>



<p class="wp-block-paragraph"><strong>Per-file overhead.</strong> Every object Athena reads costs a listing entry, an S3 GET, and header parsing before a single useful byte is decoded. On a Parquet file that overhead is relatively fixed, so as file size shrinks it becomes the dominant cost. AWS guidance targets splits of roughly 128 MB, and Parquet&#8217;s default row group size is 128 MB with ORC stripes at 64 MB. Below that, the columnar format is paying its own tax without earning it back. Splitting one hour of data across six partition levels turns a healthy 200 MB file into hundreds of files measured in kilobytes.</p>



<p class="wp-block-paragraph"><strong>S3 request rate.</strong> This is the failure mode that arrives as an actual error rather than slowness. S3 enforces a request rate ceiling per prefix, and a query fanning out across an enormous number of small objects can trip it. AWS explicitly calls out unpartitioned or badly structured data causing S3 exceptions on the GET rate. When your query fails with a throttling error rather than a timeout, look at object count, not data volume.</p>



<p class="wp-block-paragraph">Streaming ingestion is the usual source. A delivery stream buffering every sixty seconds produces well over a thousand objects per day per partition, and none of them are big. The layout is not wrong; the buffering is just not aligned with how the data is read.</p>



<h3 class="wp-block-heading">Compacting without breaking the table</h3>



<p class="wp-block-paragraph">The standard fix is a CTAS query that rewrites a slice of the table into fewer, larger objects. There is a constraint you will hit immediately if you try to do a whole table at once: Athena writes a maximum of 100 unique partition and bucket combinations per query, and exceeding it raises <code>HIVE_TOO_MANY_OPEN_PARTITIONS</code>.</p>



<p class="wp-block-paragraph">So the pattern AWS documents is CTAS to create the table with the first batch, then a series of <code>INSERT INTO</code> statements each staying under the same limit:</p>



<pre class="wp-block-code"><code>CREATE TABLE logs.access_events_compacted
WITH (
  format = 'PARQUET',
  parquet_compression = 'SNAPPY',
  partitioned_by = ARRAY['dt'],
  external_location = 's3://my-lake/access-events-compacted/'
) AS
SELECT request_id, status, path, event_time, dt
FROM logs.access_events
WHERE dt &gt;= '2024-01-01' AND dt &lt; '2024-04-01';</code></pre>



<pre class="wp-block-code"><code>INSERT INTO logs.access_events_compacted
SELECT request_id, status, path, event_time, dt
FROM logs.access_events
WHERE dt &gt;= '2024-04-01' AND dt &lt; '2024-07-01';</code></pre>



<p class="wp-block-paragraph">Two things about this that are easy to get wrong. The partition columns must come last in the select list, in the order given to <code>partitioned_by</code>. And <code>INSERT INTO</code> is not allowed against bucketed tables, so if you are also planning to bucket, you cannot use this incremental pattern for the same table.</p>



<p class="wp-block-paragraph">If compaction becomes a routine job rather than a one-off rescue, that is a strong signal to move the table to Apache Iceberg, where compaction is a maintenance operation on a live table instead of a rebuild-and-swap dance.</p>



<h2 class="wp-block-heading">Choosing partition keys without guessing</h2>



<p class="wp-block-paragraph">Most partitioning arguments are really arguments about granularity, and granularity has an arithmetic answer rather than an opinion. Work it backwards from file size.</p>



<ol class="wp-block-list">
<li><strong>Measure compressed daily volume for the table.</strong> Not raw, compressed, because that is what lands in S3.</li>

<li><strong>Divide by your target file size.</strong> Somewhere in the low hundreds of megabytes is a reasonable target. If a day of data does not comfortably exceed that, do not partition below daily. If a day is many gigabytes, hourly is defensible.</li>

<li><strong>Only add a second dimension if it appears in nearly every query.</strong> A partition key that half your queries ignore multiplies your partition count while pruning nothing for those queries. Region and tenant are the classic examples, useful when every consumer is tenant-scoped and pure overhead when they are not.</li>

<li><strong>Use bucketing for high-cardinality filters instead.</strong> Bucketing on something like a user ID gives Athena a way to skip files within a partition without creating a prefix per value.</li>

<li><strong>Sort within files on your second-most-common filter.</strong> Row group statistics only help skip data when values are clustered. Sorted data makes the min and max in the metadata meaningful; randomly ordered data makes every row group look like it might match.</li>
</ol>



<p class="wp-block-paragraph">On the recurring question of a single <code>dt=2024-05-01</code> column versus separate <code>year</code>, <code>month</code> and <code>day</code> columns: a single ISO-formatted date column is easier to filter correctly, is a clean fit for date-type projection, and makes range queries trivial. Split columns are worth it when you genuinely query at month or year granularity often enough to benefit from pruning at that level. Most tables do not, and the split scheme mostly buys you three chances to forget a predicate instead of one.</p>



<h2 class="wp-block-heading">Troubleshooting Athena partitioning problems</h2>



<p class="wp-block-paragraph">A short decision path for when something is wrong and you do not yet know which family you are in.</p>



<ul class="wp-block-list">
<li><strong>Query returns zero rows, no error.</strong> If projection is enabled, check the configured range first. Out-of-range values return empty rather than failing. Also check the format string against the actual prefix text; one is dashed and the other is not more often than you would think.</li>

<li><strong>Bytes scanned barely changes when you narrow the filter.</strong> Pruning is not happening. Look for functions, joins and subqueries on the partition column, then confirm with <code>EXPLAIN</code>.</li>

<li><strong>Query is slow but scans almost nothing.</strong> Planning cost. Consider a Glue partition index with <code>partition_filtering.enabled</code>, or partition projection.</li>

<li><strong>New data is invisible to queries.</strong> Hive-style layouts need <code>MSCK REPAIR TABLE</code>, non-Hive layouts need <code>ALTER TABLE ADD PARTITION</code>, and neither is needed with projection. Confirm which regime the table is actually in before running anything.</li>

<li><strong><code>MSCK REPAIR TABLE</code> runs clean but adds nothing.</strong> It only understands <code>key=value</code> prefixes. It also skips paths containing uppercase characters, which is a genuinely infuriating hour to lose.</li>

<li><strong>Partitions from a different table appear.</strong> <code>MSCK REPAIR TABLE</code> walks subfolders, so a table whose location sits above another table&#8217;s data will adopt its partitions. Give every table its own prefix hierarchy.</li>

<li><strong>Throttling errors from S3.</strong> Object count, not data size. Time to compact.</li>

<li><strong><code>HIVE_TOO_MANY_OPEN_PARTITIONS</code>.</strong> You crossed the write limit in a single CTAS or <code>INSERT INTO</code>. Split it into batches.</li>
</ul>



<p class="wp-block-paragraph">Worth knowing: Athena has no setting that requires queries to filter on a partition column. There is no equivalent of the partition-filter-required flag some engines expose. What you can do is put a bytes-scanned cutoff on the workgroup, so a query that ignores the partition scheme fails fast instead of running to completion.</p>



<pre class="wp-block-code"><code>aws athena create-work-group 
  --name analytics-guarded 
  --configuration '{
    "ResultConfiguration": {"OutputLocation": "s3://my-athena-results/analytics/"},
    "BytesScannedCutoffPerQuery": 10737418240,
    "EnforceWorkGroupConfiguration": true,
    "PublishCloudWatchMetricsEnabled": true
  }'</code></pre>



<p class="wp-block-paragraph">With CloudWatch metrics published per workgroup, bytes scanned becomes a graphable series. Whether you look at it in CloudWatch directly, in Amazon Managed Grafana, or in Grafana Cloud alongside the rest of your telemetry, the point is that a partitioning regression stops being invisible. Cost platforms such as Vantage or CloudZero will attribute the spend back to a workgroup for you if you want the finance-facing version of the same signal.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Partitioning by a column that queries rarely filter on, then wondering why nothing improved.</li>

<li>Adding hourly partitions to a table that receives a few megabytes an hour.</li>

<li>Filtering the timestamp column that lives inside the data instead of the partition column derived from it.</li>

<li>Enabling projection and leaving the range open-ended into the distant past, so most projected partitions point at nothing.</li>

<li>Creating Glue partition indexes and never setting <code>partition_filtering.enabled</code>, so the index sits there doing nothing.</li>

<li>Mixed-case S3 prefixes, which quietly break <code>MSCK REPAIR TABLE</code>.</li>

<li>Treating projection as a drop-in upgrade on a lake that other engines also read.</li>

<li>Compacting into the same prefix the table already points at, which is how you get duplicated rows in the middle of a rewrite.</li>
</ul>



<h2 class="wp-block-heading">Best practices for Athena partitioning</h2>



<ul class="wp-block-list">
<li>Pick granularity from compressed daily volume divided by a target file size, not from instinct.</li>

<li>Use a single ISO date partition column unless you have a concrete reason to split it.</li>

<li>Prefer partition projection for predictable time-series layouts, and Glue partition indexes for shared or irregular tables.</li>

<li>Resolve dynamic partition values in the orchestration layer and pass literals into the SQL.</li>

<li>Store data in Parquet or ORC with a splittable compression codec, and sort within files on a common secondary filter.</li>

<li>Compact streaming output on a schedule, or align the delivery buffer so it produces usable file sizes in the first place.</li>

<li>Put a bytes-scanned cutoff on every workgroup, including the one analysts use interactively.</li>

<li>Track bytes scanned per workgroup as a time series and treat a step change as an incident, not a curiosity.</li>

<li>Test partition changes against a copy first. Rewriting a table&#8217;s layout under live consumers is not something to improvise.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How many partitions is too many for one Athena table?</h3>



<p class="wp-block-paragraph">There is a hard ceiling, since Athena cannot read more than a million partitions in a single scan, but you will feel the pain long before that. The practical limit is where planning time becomes a noticeable fraction of query time. If a highly selective query spends most of its life before reading data, you are over-partitioned regardless of the absolute count.</p>



<h3 class="wp-block-heading">Does partition projection make queries faster on a table with few partitions?</h3>



<p class="wp-block-paragraph">Not meaningfully. Projection removes catalog lookup latency, and on a small table that lookup was never the problem. Its value on small tables is operational rather than performance related, since you stop having to register new partitions at all.</p>



<h3 class="wp-block-heading">Can I use partition projection and registered partitions on the same table?</h3>



<p class="wp-block-paragraph">No. Enabling projection makes Athena ignore the partition metadata registered for that table. It is one regime or the other, which matters if you have hand-registered partitions pointing at locations outside the normal pattern.</p>



<h3 class="wp-block-heading">Why did my Athena query scan the whole table even though I filtered on the partition column?</h3>



<p class="wp-block-paragraph">Almost always because the predicate could not be evaluated at planning time. A function wrapped around the partition column, a value arriving through a join, or a subquery producing the value all defeat pruning. Run <code>EXPLAIN</code> and check whether the constraint reaches the table scan node or sits in a filter above it.</p>



<h3 class="wp-block-heading">Should I partition by year, month and day or by a single date column?</h3>



<p class="wp-block-paragraph">A single ISO-formatted date column is the better default. It maps cleanly onto date-type projection, makes range filters straightforward, and gives you one predicate to remember instead of three. Split columns earn their place only if you routinely query at month or year granularity.</p>



<h3 class="wp-block-heading">Does partitioning help if my data is CSV or JSON?</h3>



<p class="wp-block-paragraph">Yes, because pruning happens at the prefix level and does not care about file format. But you are leaving the larger win on the table. Row-oriented text formats force Athena to read every column of every surviving row, so converting to Parquet or ORC typically cuts far more than partitioning alone. Do both, and do the format conversion first if you have to pick.</p>



<h3 class="wp-block-heading">How do I stop analysts from running unpartitioned queries?</h3>



<p class="wp-block-paragraph">Athena cannot require a partition filter. The workable control is a per-query bytes-scanned cutoff on the workgroup, which cancels a runaway query before it finishes. Pair it with published CloudWatch metrics so you can see who is hitting it and why.</p>



<h3 class="wp-block-heading">Is it worth moving to Iceberg instead of tuning Hive-style partitions?</h3>



<p class="wp-block-paragraph">If you are rewriting files regularly, handling late-arriving data, or fighting the write-side partition limits, Iceberg removes a lot of that work through hidden partitioning and table maintenance operations. If your table is append-only, time-partitioned and stable, Hive-style partitioning with projection is simpler and there is no prize for migrating.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Athena partitioning is not a property of your S3 layout. It is a property of the interaction between your layout, your catalog, and the exact text of your queries, and all three have to cooperate before a single byte is saved. A perfect prefix structure with a predicate the planner cannot evaluate reads the whole table. A perfect predicate against half a million tiny objects reads them all one GET at a time.</p>



<p class="wp-block-paragraph">So measure the thing that tells the truth. Bytes scanned should fall roughly in proportion to how much you narrow the filter. When it does not, one of the three reductions has stopped firing, and now you know which questions to ask to find out which one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Athena tables?</h2>



<p class="wp-block-paragraph">Most of the work here is diagnosis rather than construction. Things I help with:</p>



<ul class="wp-block-list">
<li>Auditing a slow or expensive Athena workload and identifying whether the cost is pruning, planning, or file layout.</li>

<li>Designing a partition scheme from your actual query patterns and daily volumes, including whether to bucket or sort instead.</li>

<li>Migrating tables to partition projection or Glue partition indexes without breaking the other engines reading the same lake.</li>

<li>Building compaction jobs for streaming ingestion that respect the write-side partition limits and do not duplicate data mid-run.</li>

<li>Rewriting orchestrated queries so dynamic partition values resolve to literals and actually prune.</li>

<li>Setting up workgroup guardrails and bytes-scanned monitoring so the next regression shows up on a dashboard rather than an invoice.</li>
</ul>



<p class="wp-block-paragraph">If you send me your table DDL and the <code>EXPLAIN</code> output for one slow query, I can usually tell you which of the three failure families you are in before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/athena-partitioning-performance/">Athena Partitioning: Why Your Queries Are Still Slow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</title>
		<link>https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 27 Oct 2026 06:05:56 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon DataZone]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudFormation]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[IAM Identity Center]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[SageMaker AI]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=513</guid>

					<description><![CDATA[<p>SageMaker Unified Studio is a front door, not a runtime. It consolidates the interface while leaving your bill, your IAM design and your infrastructure as code exactly where they were, and it quietly hands you a DataZone governance system to operate. Here is what it genuinely replaces, what it doesn't, the failure modes that bite during adoption, and how to decide whether it is worth it for your team.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question always arrives in the same shape. A team stands up a domain, everyone logs in, the demo goes well, and then somebody asks when they can turn off the Glue console and the Athena query editor.</p>



<p class="wp-block-paragraph">They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.</p>



<p class="wp-block-paragraph">This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.</p>



<h2 class="wp-block-heading">SageMaker Unified Studio is a front door, not a runtime</h2>



<p class="wp-block-paragraph">Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.</p>



<p class="wp-block-paragraph">AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project&#8217;s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new <em>authorisation</em> layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.</p>



<h2 class="wp-block-heading">What it actually replaces</h2>



<h3 class="wp-block-heading">The pile of standalone studios and query editors</h3>



<p class="wp-block-paragraph">This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.</p>



<p class="wp-block-paragraph">The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a &#8220;workspace&#8221; meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.</p>



<h3 class="wp-block-heading">Per-team domain sprawl</h3>



<p class="wp-block-paragraph">The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.</p>



<h3 class="wp-block-heading">The access request spreadsheet</h3>



<p class="wp-block-paragraph">There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.</p>



<p class="wp-block-paragraph">The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.</p>



<h2 class="wp-block-heading">What it doesn&#8217;t replace</h2>



<h3 class="wp-block-heading">Your bill, or your ability to explain it</h3>



<p class="wp-block-paragraph">There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.</p>



<p class="wp-block-paragraph">What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer&#8217;s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.</p>



<p class="wp-block-paragraph">Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.</p>



<h3 class="wp-block-heading">Your IAM and Lake Formation design</h3>



<p class="wp-block-paragraph">Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.</p>



<p class="wp-block-paragraph">Nobody in the adoption meeting will say &#8220;this fixes our permissions mess.&#8221; Plenty of people will quietly assume it. Say it out loud early.</p>



<h3 class="wp-block-heading">Your infrastructure as code</h3>



<p class="wp-block-paragraph">This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.</p>



<p class="wp-block-paragraph">It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.</p>



<h3 class="wp-block-heading">SageMaker AI</h3>



<p class="wp-block-paragraph">The original SageMaker was renamed SageMaker AI and remains available on its own. If your team&#8217;s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.</p>



<h2 class="wp-block-heading">What SageMaker Unified Studio quietly adds</h2>



<p class="wp-block-paragraph">Here is the section that rarely makes it into the migration plan.</p>



<h3 class="wp-block-heading">You now operate DataZone, intentionally or not</h3>



<p class="wp-block-paragraph">SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a <code>dzd</code> prefix, and the console for creating one still lives under the DataZone console path.</p>



<p class="wp-block-paragraph">You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:</p>



<pre class="wp-block-code"><code># Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID</code></pre>



<p class="wp-block-paragraph">Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.</p>



<h3 class="wp-block-heading">Project creation is a CloudFormation deploy wearing a friendly hat</h3>



<p class="wp-block-paragraph">Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.</p>



<p class="wp-block-paragraph">This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation &#8220;just doesn&#8217;t work,&#8221; do not reproduce it in the UI. Read the stack events:</p>



<pre class="wp-block-code"><code># Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The first failed resource causes the rollback. Everything after it is noise. This one habit turns most &#8220;the platform is broken&#8221; tickets into five-minute answers.</p>



<h3 class="wp-block-heading">A domain-type fork you will live with for a long time</h3>



<p class="wp-block-paragraph">Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.</p>



<p class="wp-block-paragraph">Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post&#8217;s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.</p>



<h2 class="wp-block-heading">Where adoption actually goes wrong</h2>



<h3 class="wp-block-heading">&#8220;It works in my console but not in the project&#8221;</h3>



<p class="wp-block-paragraph">This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.</p>



<p class="wp-block-paragraph">Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:</p>



<pre class="wp-block-code"><code>aws sts get-caller-identity</code></pre>



<p class="wp-block-paragraph">If the ARN is not the one you assumed, stop. You have found the problem.</p>



<h3 class="wp-block-heading">The bill went up after a migration that was supposed to save money</h3>



<p class="wp-block-paragraph">Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.</p>



<h3 class="wp-block-heading">Nobody can find the data that is definitely there</h3>



<p class="wp-block-paragraph">Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a migration rather than an addition.</strong> Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.</li>
<li><strong>Letting users self-serve projects on day one.</strong> Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.</li>
<li><strong>Choosing the domain authentication type in a hurry.</strong> It is the hardest thing here to change later.</li>
<li><strong>Skipping the catalog.</strong> Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.</li>
<li><strong>Assuming the CI/CD CLI provisions your platform.</strong> It deploys applications into projects that already exist.</li>
<li><strong>Not tagging project resources from the start.</strong> Retrofitting attribution across dozens of blueprint-created resources is miserable work.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">A rough order of operations for anyone weighing this up:</p>



<ol class="wp-block-list">
<li><strong>Ask what problem you are solving.</strong> If it is &#8220;our teams cannot discover or safely share each other&#8217;s data,&#8221; this is the right tool. If it is &#8220;we have too many consoles,&#8221; it is an expensive way to buy fewer bookmarks.</li>
<li><strong>Count your teams.</strong> The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.</li>
<li><strong>Decide the authentication model first.</strong> Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.</li>
<li><strong>Write the project profiles yourself.</strong> The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.</li>
<li><strong>Prove the teardown path before the build path.</strong> Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.</li>
<li><strong>Keep the old consoles.</strong> They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.</li>
</ol>



<p class="wp-block-paragraph">One more piece of judgement: wire your existing observability in from the start rather than relying on the studio&#8217;s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Unified Studio replace SageMaker Studio?</h3>



<p class="wp-block-paragraph">It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.</p>



<h3 class="wp-block-heading">Do I still need Amazon DataZone separately?</h3>



<p class="wp-block-paragraph">Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.</p>



<h3 class="wp-block-heading">How much does SageMaker Unified Studio cost?</h3>



<p class="wp-block-paragraph">The interface has no standalone charge. You pay for each AWS service you use through it on that service&#8217;s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.</p>



<h3 class="wp-block-heading">Can I manage domains and projects with Terraform?</h3>



<p class="wp-block-paragraph">AWS&#8217;s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.</p>



<h3 class="wp-block-heading">What happens to my Glue jobs and Athena queries?</h3>



<p class="wp-block-paragraph">Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.</p>



<h3 class="wp-block-heading">Is it worth adopting for a small team?</h3>



<p class="wp-block-paragraph">Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.</p>



<h3 class="wp-block-heading">Can I run both the old and new experiences at once?</h3>



<p class="wp-block-paragraph">Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.</p>



<p class="wp-block-paragraph">That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was &#8220;fewer tabs.&#8221; The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.</p>



<p class="wp-block-paragraph">Ask which one you are before the first project gets created, not after the fortieth.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your adoption plan?</h2>



<p class="wp-block-paragraph">This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup</li>
<li>Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults</li>
<li>Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use</li>
<li>Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover</li>
<li>Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants</li>
<li>Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long</li>
</ul>



<p class="wp-block-paragraph">Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 26 Oct 2026 06:37:42 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Machine Learning]]></category>
		<category><![CDATA[Retail Technology]]></category>
		<category><![CDATA[Amazon Personalize]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Clickstream Ingestion]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Idle Resources]]></category>
		<category><![CDATA[Item Cold Start]]></category>
		<category><![CDATA[Kinesis Data Streams]]></category>
		<category><![CDATA[Model Serving]]></category>
		<category><![CDATA[Recommendation Systems]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Training-Serving Skew]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=894</guid>

					<description><![CDATA[<p>An idle Amazon Personalize setup does not throw errors. It just bills. A practical walkthrough of the five decisions that shape an AWS ecommerce recommendation pipeline: domain recommenders versus custom recipes, bulk versus streaming ingestion, schema limits, training cadence, and real-time versus batch serving, with the failure modes that stay invisible until the invoice arrives.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually arrives from finance rather than from engineering. Something along the lines of: why is there an Amazon Personalize charge on an account where nobody has shipped a recommendation carousel yet?</p>



<p class="wp-block-paragraph">The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.</p>



<p class="wp-block-paragraph">I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.</p>



<p class="wp-block-paragraph">This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.</p>



<h2 class="wp-block-heading">The bill arrives before the carousel does</h2>



<p class="wp-block-paragraph">Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.</p>



<p class="wp-block-paragraph"><strong>Solution training.</strong> New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.</p>



<p class="wp-block-paragraph"><strong>Inference capacity.</strong> A campaign has <code>minProvisionedTPS</code>, a domain recommender has <code>minRecommendationRequestsPerSecond</code>. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.</p>



<p class="wp-block-paragraph">Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])</code></pre>



<h2 class="wp-block-heading">Decision one: domain recommender or custom solution</h2>



<p class="wp-block-paragraph">A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.</p>



<p class="wp-block-paragraph">The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.</p>



<p class="wp-block-paragraph">Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.</p>



<p class="wp-block-paragraph">Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.</p>



<p class="wp-block-paragraph">On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.</p>



<p class="wp-block-paragraph">One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at <code>aws-hrnn</code>, its advice about cold items and training cadence is stale too.</p>



<h2 class="wp-block-heading">Decision two: how interaction data gets in</h2>



<h3 class="wp-block-heading">Bulk imports replace unless you tell them not to</h3>



<p class="wp-block-paragraph">This is the one that costs people a month of confused model quality.</p>



<p class="wp-block-paragraph">A dataset import job defaults to <code>FULL</code>, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is <code>INCREMENTAL</code>, which appends and only replaces records sharing an ID.</p>



<p class="wp-block-paragraph">Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)</code></pre>



<p class="wp-block-paragraph">Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.</p>



<h3 class="wp-block-heading">Streaming events through an event tracker</h3>



<p class="wp-block-paragraph">The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the <code>personalize-events</code> client.</p>



<pre class="wp-block-code"><code>import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)</code></pre>



<p class="wp-block-paragraph">Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.</p>



<p class="wp-block-paragraph">That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.</p>



<p class="wp-block-paragraph">The <code>sessionId</code> field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.</p>



<h2 class="wp-block-heading">Decision three: schema and event types, decided once</h2>



<p class="wp-block-paragraph">Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.</p>



<pre class="wp-block-code"><code>{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}</code></pre>



<p class="wp-block-paragraph">The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.</p>



<p class="wp-block-paragraph">Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.</p>



<p class="wp-block-paragraph">Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.</p>



<p class="wp-block-paragraph">On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.</p>



<p class="wp-block-paragraph">Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.</p>



<h2 class="wp-block-heading">Decision four: freshness without a retraining bill</h2>



<p class="wp-block-paragraph">Two mechanisms get conflated constantly, and the difference sets your monthly cost.</p>



<p class="wp-block-paragraph"><strong>Automatic training</strong> creates a new solution version. The model learns from your latest data. It costs a training run each time.</p>



<p class="wp-block-paragraph"><strong>Automatic updates</strong> are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.</p>



<p class="wp-block-paragraph">So automatic updates solve the cold item problem and do nothing about model drift. AWS&#8217;s own guidance is that even with them, training cadence should stay at least weekly.</p>



<p class="wp-block-paragraph">Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.</p>



<p class="wp-block-paragraph">The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.</p>



<h2 class="wp-block-heading">Decision five: how the recommendation reaches the page</h2>



<p class="wp-block-paragraph">Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.</p>



<p class="wp-block-paragraph">Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.</p>



<p class="wp-block-paragraph">Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.</p>



<p class="wp-block-paragraph">Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.</p>



<h2 class="wp-block-heading">Troubleshooting an AWS ecommerce recommendation pipeline</h2>



<ul class="wp-block-list">
<li><strong>Everyone gets the same recommendations.</strong> Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.</li>



<li><strong>New products never show up.</strong> Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.</li>



<li><strong>Training fails on data volume.</strong> Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.</li>



<li><strong>The dataset shrank and nothing errored.</strong> A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.</li>



<li><strong>Recommendations ignore what the user just clicked.</strong> Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.</li>



<li><strong>The bill rose during a quiet week.</strong> Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.</li>



<li><strong>Filters are not excluding purchased items.</strong> Filtering looks at a bounded slice of each user&#8217;s history per event type. Heavy users push older purchases outside it.</li>
</ul>



<p class="wp-block-paragraph">Auditing an account you did not build starts with listing what exists, not with reading the application code.</p>



<pre class="wp-block-code"><code>aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN</code></pre>



<p class="wp-block-paragraph">Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving automatic training on for a solution that exists only to satisfy curiosity.</li>



<li>Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.</li>



<li>Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.</li>



<li>Recording events only for authenticated users, starving the model of most of your traffic.</li>



<li>Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.</li>



<li>Calling PutEvents synchronously on the render path.</li>



<li>Following an implementation guide that still references the retired HRNN recipes.</li>



<li>Shipping with no fallback, so the first slow inference call becomes a slow product page.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.</li>



<li>Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.</li>



<li>Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.</li>



<li>Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.</li>



<li>Prefer batch inference plus a cache for anything that does not have to react within the session.</li>



<li>Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.</li>



<li>Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much data do I need before Personalize is worth it?</h3>



<p class="wp-block-paragraph">The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.</p>



<h3 class="wp-block-heading">Can I build this without Amazon Personalize?</h3>



<p class="wp-block-paragraph">Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.</p>



<h3 class="wp-block-heading">What does an idle Personalize setup cost?</h3>



<p class="wp-block-paragraph">More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.</p>



<h3 class="wp-block-heading">Real-time or batch recommendations?</h3>



<p class="wp-block-paragraph">Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.</p>



<h3 class="wp-block-heading">Do new products appear automatically?</h3>



<p class="wp-block-paragraph">Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.</p>



<h3 class="wp-block-heading">How do I connect a Shopify or WooCommerce store?</h3>



<p class="wp-block-paragraph">There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.</p>



<h3 class="wp-block-heading">How often should I retrain?</h3>



<p class="wp-block-paragraph">At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.</p>



<p class="wp-block-paragraph">Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your recommendation pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for ecommerce teams building on AWS:</p>



<ul class="wp-block-list">
<li>Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.</li>



<li>Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.</li>



<li>Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.</li>



<li>Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.</li>



<li>Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.</li>



<li>Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.</li>
</ul>



<p class="wp-block-paragraph">If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</title>
		<link>https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 25 Oct 2026 06:06:42 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Earned Schedule]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Last Planner System]]></category>
		<category><![CDATA[Leading Indicators]]></category>
		<category><![CDATA[Owner Dashboard]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Procore]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Total Float]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1033</guid>

					<description><![CDATA[<p>Schedule delay rarely arrives as a surprise. It sits in float erosion, aging RFIs and slipping weekly commitments long before the monthly update moves the finish date. Here is how to build a dashboard that catches it while there is still float left to spend.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly schedule update lands on a Thursday. The planner has re-run the logic, and substantial completion has moved three weeks to the right. The project manager asks the obvious question in the next meeting: why is this the first time anyone is hearing about it?</p>



<p class="wp-block-paragraph">The uncomfortable answer is that people did know. The superintendent knew the drywall crew had been short for two weeks. The project engineer knew a structural RFI had been sitting with the design team well past its due date. The foreman knew the weekly commitments kept slipping. None of it was hidden. It just never reached one screen, in one place, while there was still float left to spend.</p>



<p class="wp-block-paragraph">This post walks through building a <strong>construction early warning dashboard</strong> that catches those signals while they are still cheap to act on. It covers why schedule delay shows up late, the one data decision that makes early warning possible, the signals worth tracking, how to pull data out of Primavera P6, Microsoft Project and Procore, and how to alert without training everyone to ignore the alerts.</p>



<h2 class="wp-block-heading">Why Construction Delay Shows Up Three Weeks Late</h2>



<p class="wp-block-paragraph">Delay rarely arrives as one event. It accumulates in small amounts across activities that do not look critical yet. Three things keep it invisible.</p>



<ol class="wp-block-list">
<li><strong>The schedule is updated in batches.</strong> If the CPM schedule is statused monthly, or even biweekly, the finish date is only recalculated at that cadence. Everything that happened in between lands at once.</li>



<li><strong>Progress is self-reported.</strong> Percent complete is an estimate from whoever statuses the activity. Optimism is not dishonesty, but it compounds. An activity that sits at a comfortable-looking percentage for three updates is telling you something.</li>



<li><strong>The data is overwritten in place.</strong> This is the one that bites hardest. Most schedule tools, and most dashboards built on top of them, show the current value of float, the current forecast finish and the current status. Last month&#8217;s values are gone unless someone saved the file.</li>
</ol>



<p class="wp-block-paragraph">That third point is why so many project dashboards look fine right up until they don&#8217;t. A current-state dashboard tells you where you are. Delay warnings live in the rate of change. An activity with twelve days of float looks healthy. The same activity with twelve days of float, down from thirty over the last two updates, is the next critical path.</p>



<h2 class="wp-block-heading">The Design Decision That Makes Early Warning Possible: Snapshot Everything</h2>



<p class="wp-block-paragraph">Before any chart or alert, decide that every schedule update, every RFI log pull and every weekly plan gets stored as a dated snapshot. You append. You never overwrite. Storage for this is trivial; a few hundred activities per update is nothing for any database.</p>



<p class="wp-block-paragraph">PostgreSQL is what I reach for first here. It runs happily on a small managed instance like Amazon RDS or DigitalOcean Managed Databases, or on a modest VPS from Hetzner, Contabo or Interserver if you are comfortable handling backups yourself. A minimal activity snapshot table looks like this:</p>



<pre class="wp-block-code"><code>CREATE TABLE activity_snapshot (
    project_code     text        NOT NULL,
    data_date        date        NOT NULL,
    task_code        text        NOT NULL,
    task_name        text,
    total_float_days numeric,
    early_start      date,
    early_finish     date,
    is_complete      boolean     NOT NULL DEFAULT false,
    loaded_at        timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_code, data_date, task_code)
);</code></pre>



<p class="wp-block-paragraph">The primary key does the important work. Each activity appears once per data date, so loading the same export twice fails loudly instead of silently doubling the history. The <code>data_date</code> is the schedule&#8217;s status date, not the day you loaded the file, because trends must be plotted against when the schedule says it was true.</p>



<p class="wp-block-paragraph">Keying on <code>task_code</code>, the activity ID the planner sees, rather than an internal database ID matters too. Internal IDs can change when a schedule is re-imported. Activity IDs usually survive, and when they don&#8217;t, you want to know (more on that in troubleshooting).</p>



<h2 class="wp-block-heading">Early Warning Signals That Move Before the Finish Date Does</h2>



<p class="wp-block-paragraph">Each of these is a leading indicator of schedule delay. None is perfect alone. Together they give you weeks of notice instead of a surprise in the monthly update.</p>



<h3 class="wp-block-heading">Float Erosion on Near-Critical Paths</h3>



<p class="wp-block-paragraph">Watching the critical path is necessary but late. By the time an activity is critical, its float is already gone. The useful signal is near-critical work losing float quickly. This query compares each open activity to its own previous snapshot:</p>



<pre class="wp-block-code"><code>WITH history AS (
    SELECT
        project_code,
        task_code,
        task_name,
        data_date,
        total_float_days,
        LAG(total_float_days) OVER (
            PARTITION BY project_code, task_code
            ORDER BY data_date
        ) AS prev_float_days
    FROM activity_snapshot
    WHERE NOT is_complete
)
SELECT
    project_code,
    task_code,
    task_name,
    prev_float_days,
    total_float_days,
    prev_float_days - total_float_days AS float_lost_days
FROM history
WHERE data_date = (SELECT max(data_date) FROM activity_snapshot)
  AND prev_float_days IS NOT NULL
  AND total_float_days &lt;= 20
  AND prev_float_days - total_float_days &gt;= 5
ORDER BY float_lost_days DESC;</code></pre>



<p class="wp-block-paragraph">The thresholds (twenty days remaining, five days lost) are starting points, not rules. Tune them to your update cadence and project length. What matters is that the query looks at movement, which a current-state report cannot.</p>



<h3 class="wp-block-heading">Constraint Aging: RFIs and Submittals Against Need-By Dates</h3>



<p class="wp-block-paragraph">An overdue RFI on its own is noise. Every project has them. An overdue RFI that blocks an activity starting in ten days is a delay you can see coming.</p>



<p class="wp-block-paragraph">The hard part is the link. RFI and submittal logs in Procore or Autodesk Construction Cloud rarely point at schedule activity IDs. Someone has to maintain a small mapping table from RFI or submittal to activity, or at least to area and spec section. I would not skip it. Without the link you get a list of late paperwork; with it you get &#8220;this answer is needed in eight days and the ball has been in the architect&#8217;s court for three weeks.&#8221;</p>



<p class="wp-block-paragraph">The signal to chart is days of slack between the expected response and the need-by date of the linked activity. When it goes negative, the RFI is now on the schedule&#8217;s path whether the schedule knows it or not.</p>



<h3 class="wp-block-heading">Commitment Reliability: PPC and Reasons for Variance</h3>



<p class="wp-block-paragraph">If the field runs the Last Planner System, you already have a strong signal. Percent Plan Complete (PPC) is the share of weekly commitments finished as promised. Each week&#8217;s PPC is a lagging number, but the trend across weeks leads the CPM schedule, because missed promises show up in the field long before they show up as recalculated dates.</p>



<p class="wp-block-paragraph">Store reasons for variance as categories, not free text. A PPC dip caused by weather tells you something different from three straight weeks of &#8220;material not on site.&#8221; The second one is a procurement problem that will keep producing delay until someone fixes it.</p>



<h3 class="wp-block-heading">Look-Ahead Churn</h3>



<p class="wp-block-paragraph">Count how many times an activity&#8217;s planned start has moved across snapshots of the three- or six-week look-ahead. An activity pushed out once is normal planning. One that has been pushed four updates running is being deferred, not planned, and the reason is usually a constraint nobody has named yet.</p>



<h3 class="wp-block-heading">Earned Value, Read Carefully</h3>



<p class="wp-block-paragraph">If you track earned value, the Schedule Performance Index (SPI = EV / PV) belongs on the dashboard, with a caveat many dashboards skip. Classic SPI drifts back toward 1.0 as the project nears completion, even when the job is late, because earned value and planned value both end at the budget at completion. A late-stage SPI that looks like recovery may just be arithmetic.</p>



<p class="wp-block-paragraph">Earned Schedule fixes this by measuring in time rather than money. It asks at what point in the baseline your current earned value was supposed to be reached, then divides that by the actual time elapsed to get SPI(t). It needs no new data collection, only different math on the same numbers. If you only plot one schedule index late in a job, plot SPI(t).</p>



<h3 class="wp-block-heading">Claimed Progress Versus Manpower</h3>



<p class="wp-block-paragraph">Daily logs record who was on site. When an activity&#8217;s reported percent complete keeps rising while the manpower logged against that trade stays well below the plan, one of those numbers is wrong. Usually it is the percent complete. This comparison is crude, and it needs a consistent trade mapping, but it is one of the few checks that uses data nobody is trying to make look good.</p>



<h2 class="wp-block-heading">Getting the Data Out of P6, Microsoft Project and Procore</h2>



<p class="wp-block-paragraph">A construction project dashboard lives or dies on boring extraction work. Plan for it.</p>



<h3 class="wp-block-heading">Primavera P6 XER Exports</h3>



<p class="wp-block-paragraph">An XER file is plain tab-delimited text. Lines beginning with <code>%T</code> name a table, <code>%F</code> lists its fields, and <code>%R</code> lines are rows. Activities live in the <code>TASK</code> table. This reader pulls one table into a list of dictionaries:</p>



<pre class="wp-block-code"><code>def read_xer_table(path, table_name):
    rows, fields, current = [], None, None
    # XER exports often use a Windows code page, not UTF-8
    with open(path, encoding="cp1252", errors="replace") as fh:
        for line in fh:
            parts = line.rstrip("rn").split("t")
            tag = parts[0]
            if tag == "%T":
                current = parts[1]
            elif tag == "%F" and current == table_name:
                fields = parts[1:]
            elif tag == "%R" and current == table_name:
                rows.append(dict(zip(fields, parts[1:])))
    return rows

tasks = read_xer_table("update.xer", "TASK")
for t in tasks[:5]:
    print(t["task_code"], t["task_name"], t["total_float_hr_cnt"])</code></pre>



<p class="wp-block-paragraph">Two details catch people. Float and durations in XER are stored in hours (<code>total_float_hr_cnt</code>), so converting to days means dividing by the hours per day of each activity&#8217;s calendar, not a blanket eight. And an activity with a value in <code>act_end_date</code> has an actual finish, which is a clean way to set <code>is_complete</code> without relying on status labels.</p>



<p class="wp-block-paragraph">Use the float P6 stored in the export rather than recomputing the critical path yourself. P6&#8217;s result reflects its scheduling options, constraints and calendars. Your recompute probably won&#8217;t.</p>



<h3 class="wp-block-heading">Microsoft Project</h3>



<p class="wp-block-paragraph">Microsoft Project can save a schedule as XML, which carries tasks, dates and total slack. The same snapshot logic applies. Save one file per status date, load it with its data date, and never replace the previous one.</p>



<h3 class="wp-block-heading">Procore and Other Project Management Platforms</h3>



<p class="wp-block-paragraph">Procore&#8217;s REST API exposes RFIs and submittals per project, including status, due dates and ball-in-court. Two warnings. Most useful endpoints are scoped to a single project, so a portfolio view means iterating over the project list. And submittal status values can be configured per company, so map them explicitly rather than hard-coding what you saw on one account. Autodesk Construction Cloud has its own RFI API with a similar shape.</p>



<h3 class="wp-block-heading">The Spreadsheet Nobody Wants to Admit Exists</h3>



<p class="wp-block-paragraph">The weekly work plan is very often an Excel file or a Smartsheet. Accept that. A scheduled job that reads a fixed-layout sheet from a shared folder is less elegant than an API and far more likely to get used. Fighting the field&#8217;s tools is how early warning systems die.</p>



<h2 class="wp-block-heading">Building the Dashboard and Alerts People Don&#8217;t Mute</h2>



<p class="wp-block-paragraph">For visualization I lean toward Grafana, self-hosted or on Grafana Cloud, because its alert rules can run SQL directly against PostgreSQL and route results to email, Slack or Microsoft Teams. You can see the style of dashboard in this <a href="https://monitor.john-nessime.com/public-dashboards/4c1463601230401ab55159059364b739" target="_blank" rel="noreferrer noopener">live Grafana demo</a>.</p>



<p class="wp-block-paragraph">Power BI deserves a fair hearing. Many contractors already license it, project controls teams know it, and its reporting and drill-down are excellent for monthly reviews. Where it is weaker is rule-based alerting on arbitrary queries. If your organization lives in Microsoft 365, a sensible split is Power BI for the reporting pack and a small scheduled job or Grafana instance for the alerts. Metabase is another reasonable middle ground for smaller teams.</p>



<p class="wp-block-paragraph">Whatever the tool, the alert design matters more than the charts:</p>



<ul class="wp-block-list">
<li><strong>Send a weekly digest, not a page.</strong> Schedule risk is not an outage. A Monday summary of every signal that crossed a threshold gets read. Twenty individual notifications get filtered.</li>



<li><strong>Name the owner in the alert.</strong> &#8220;RFI 142 blocks level 3 slab pour, need-by in 8 days, ball in court: structural engineer&#8221; is actionable. &#8220;Constraint risk: high&#8221; is not.</li>



<li><strong>Compare the project to itself.</strong> Portfolio-wide thresholds punish complex jobs and let simple ones coast. A drop from its own recent average is the better trigger.</li>



<li><strong>Alert on stale data.</strong> If no schedule update has loaded in longer than your cadence, that is itself a warning. An early warning dashboard showing old data is worse than none.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Float values look wrong compared to P6</h3>



<p class="wp-block-paragraph">Check the hours-to-days conversion first. Then check which float definition the schedule uses. A project-level &#8220;must finish by&#8221; date or activity constraints can produce negative float that looks like a bug but is correct. Also confirm whether the planner defines critical as longest path or as float below a threshold.</p>



<h3 class="wp-block-heading">Trend lines break after a re-baseline or renumbering</h3>



<p class="wp-block-paragraph">If activity IDs change, <code>LAG()</code> finds no previous row and the activity appears brand new. Add a check that counts activities present in the previous snapshot but missing from the current one. A sudden jump means the history needs a mapping table, not a fresh start.</p>



<h3 class="wp-block-heading">PPC sits at or near 100 percent every week</h3>



<p class="wp-block-paragraph">That usually means commitments are being made at the end of the week, or tasks are so loose they cannot fail. A PPC that never moves carries no signal. Look at how commitments are recorded before trusting the number.</p>



<h3 class="wp-block-heading">The dashboard is green but the field says you are behind</h3>



<p class="wp-block-paragraph">Check how percent complete is calculated. P6 supports duration, physical and units percent complete types, and they can tell very different stories about the same activity. Believe the field, then find which input is out of step.</p>



<h2 class="wp-block-heading">Common Mistakes</h2>



<ul class="wp-block-list">
<li>Building on the live schedule only, with no history, so every trend has to be reconstructed by hand.</li>



<li>Watching only the critical path and missing the near-critical work that becomes critical next month.</li>



<li>Trusting classic SPI in the final third of a project, when it drifts toward 1.0 regardless of performance.</li>



<li>Tracking overdue RFIs without linking them to the activities they block.</li>



<li>Recomputing float yourself instead of using what the scheduling tool calculated.</li>



<li>Sending so many alerts that the project team creates an inbox rule to hide them.</li>
</ul>



<h2 class="wp-block-heading">Best Practices</h2>



<ul class="wp-block-list">
<li>Append every export with its data date. Never overwrite history.</li>



<li>Start with two signals, float erosion and constraint aging, and add more once those are trusted.</li>



<li>Agree with the planner on update cadence. The dashboard cannot be fresher than its inputs.</li>



<li>Keep the RFI-to-activity mapping small and owned by one named person.</li>



<li>Review thresholds after the first month. Early numbers are guesses.</li>



<li>Show each signal with its source and data date, so nobody argues about whether it is current.</li>
</ul>



<h2 class="wp-block-heading">Frequently Asked Questions</h2>



<h3 class="wp-block-heading">What are leading indicators of construction delay?</h3>



<p class="wp-block-paragraph">The most useful are float erosion on near-critical activities, RFIs and submittals aging toward the start of the work they block, a falling PPC trend, activities repeatedly pushed out of the look-ahead, and reported progress outpacing logged manpower.</p>



<h3 class="wp-block-heading">How often should the schedule be updated for early warning to work?</h3>



<p class="wp-block-paragraph">More often than monthly if you can manage it. Weekly or biweekly updates give the trend queries enough points to show movement. Signals from RFIs and weekly plans can refresh daily regardless of schedule cadence.</p>



<h3 class="wp-block-heading">Is SPI a good early warning metric?</h3>



<p class="wp-block-paragraph">Early and mid-project, yes, alongside float and critical path analysis. Late in a project, classic SPI drifts toward 1.0 even on late jobs, so the time-based SPI(t) from Earned Schedule is more trustworthy.</p>



<h3 class="wp-block-heading">Can I build this with Power BI instead of Grafana?</h3>



<p class="wp-block-paragraph">Yes for the reporting side, and it is often the better choice where the team already uses it. The snapshot database and trend logic are the same either way. Rule-based alerting is where you may want a separate tool.</p>



<h3 class="wp-block-heading">Do I need Primavera P6 for a construction early warning dashboard?</h3>



<p class="wp-block-paragraph">No. Any scheduling tool that exports activities with dates and float works, including Microsoft Project. What you need is a consistent export on every update and somewhere to keep the history.</p>



<h3 class="wp-block-heading">How do I link RFIs to schedule activities?</h3>



<p class="wp-block-paragraph">Usually with a small mapping table maintained by the project engineer: RFI or submittal number, activity ID, and need-by date. Mapping by area and spec section is a workable first step if activity-level links are too much at the start.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The project manager who learns about delay three weeks late usually isn&#8217;t short of data. The problem is that the data is spread across a schedule, an RFI log and a weekly plan, and each one only remembers its latest state. A construction early warning dashboard works because it keeps the history and watches the movement: float draining from work that isn&#8217;t critical yet, answers arriving after the work that needs them, promises slipping week after week. Snapshot first. Everything else is a query on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need Help Building a Construction Early Warning Dashboard?</h2>



<p class="wp-block-paragraph">I build and host the data side of this for construction teams and project controls groups. Here is where I can help:</p>



<ul class="wp-block-list">
<li>Setting up a snapshot database that stores every P6 or Microsoft Project update with its data date.</li>



<li>Writing XER and XML loaders that handle hours-based float, calendars and renumbered activities.</li>



<li>Pulling RFIs and submittals from Procore or Autodesk Construction Cloud and linking them to activities.</li>



<li>Building Grafana dashboards for float erosion, constraint aging, PPC and SPI(t) trends.</li>



<li>Designing weekly digest alerts with named owners, sent to email, Slack or Teams.</li>



<li>Hosting and maintaining the dashboard so your team reads it instead of running it.</li>
</ul>



<p class="wp-block-paragraph">If you want to scope it first, the <a href="https://john-nessime.com/tools/engagement-builder/">Engagement Builder</a> lays out phases and effort. Or send me one recent XER export and your RFI log, and I will tell you which signals your data can already support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/hosted-monitoring/">See Hosted Monitoring</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: What Each Format Actually Costs You</title>
		<link>https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 24 Oct 2026 06:02:47 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Compression]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CSV]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[JSON]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=339</guid>

					<description><![CDATA[<p>Format choice shows up as a bill, not a performance complaint. A working breakdown of where the money actually goes across Parquet, JSON and CSV: column pruning, row-group skipping, the gzip splittability trap, and request overhead. Includes a procedure for measuring the difference on your own data, and an honest account of where Parquet costs you.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;Athena spend tripled last month, nothing changed.&#8221; Nothing had changed on the query side. What changed was that an upstream team started dropping a wider export into the same S3 prefix, still as gzipped CSV, and every dashboard query that touched that table started reading every column of every file to answer a question about two of them.</p>



<p class="wp-block-paragraph">That is the failure mode that makes this comparison worth writing about. The <strong>Parquet vs JSON vs CSV</strong> question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.</p>



<p class="wp-block-paragraph">This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else&#8217;s benchmark, and an honest look at where CSV and JSON still beat the columnar option.</p>



<h2 class="wp-block-heading">The billing model is the whole story</h2>



<p class="wp-block-paragraph">Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.</p>



<p class="wp-block-paragraph">Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery&#8217;s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.</p>



<p class="wp-block-paragraph">That last part catches people. <code>LIMIT 10</code> does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which format is smaller on disk.&#8221; It is &#8220;how few bytes can the engine get away with reading.&#8221; Those diverge fast.</p>



<h2 class="wp-block-heading">Four levers, not one</h2>



<p class="wp-block-paragraph">When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.</p>



<h3 class="wp-block-heading">1. Column pruning</h3>



<p class="wp-block-paragraph">Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.</p>



<h3 class="wp-block-heading">2. Block skipping</h3>



<p class="wp-block-paragraph">Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.</p>



<p class="wp-block-paragraph">The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group&#8217;s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.</p>



<h3 class="wp-block-heading">3. Splittability</h3>



<p class="wp-block-paragraph">This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.</p>



<p class="wp-block-paragraph">Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.</p>



<p class="wp-block-paragraph">If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.</p>



<h3 class="wp-block-heading">4. Request overhead</h3>



<p class="wp-block-paragraph">Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine&#8217;s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">CSV: still the right answer more often than people admit</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.</p>



<p class="wp-block-paragraph">CSV is an excellent transport format and a poor storage format. That distinction is the useful one.</p>



<h2 class="wp-block-heading">JSON: the most expensive way to store a table</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Every record repeats every key name. Store ten million events with a field called <code>customer_identifier</code> and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.</p>



<p class="wp-block-paragraph">JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.</p>



<h2 class="wp-block-heading">Parquet: cheap reads, real obligations</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> This is the part the enthusiastic posts skip.</p>



<ul class="wp-block-list"><li><strong>You cannot append.</strong> Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.</li><li><strong>Schema mismatches across files are a runtime error, not a warning.</strong> Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.</li><li><strong>Timestamps are a compatibility minefield.</strong> The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.</li><li><strong>It is opaque.</strong> You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.</li><li><strong>Writing costs more.</strong> Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.</li></ul>



<p class="wp-block-paragraph">None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.</p>



<h2 class="wp-block-heading">Measure it on your data, in about twenty minutes</h2>



<p class="wp-block-paragraph">Published benchmarks tell you about somebody else&#8217;s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.</p>



<ol class="wp-block-list"><li><strong>Get a real byte count for what you have now.</strong> Not an estimate, the actual prefix.</li></ol>



<pre class="wp-block-code"><code>aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable</code></pre>



<p class="wp-block-paragraph">The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.</p>



<ol start="2" class="wp-block-list"><li><strong>Convert a representative slice, not the whole thing.</strong> One partition is enough. In Athena, a CTAS statement writes the output for you.</li></ol>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';</code></pre>



<p class="wp-block-paragraph">Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.</p>



<ol start="3" class="wp-block-list"><li><strong>Run your three most common queries against both tables and compare bytes scanned.</strong> Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.</li><li><strong>On BigQuery, price it before you run it.</strong> A dry run costs nothing and returns the byte estimate.</li></ol>



<pre class="wp-block-code"><code>bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'</code></pre>



<ol start="5" class="wp-block-list"><li><strong>Inspect the Parquet file you just wrote.</strong> Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.</li></ol>



<pre class="wp-block-code"><code>python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"</code></pre>



<p class="wp-block-paragraph">That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.</p>



<ol start="6" class="wp-block-list"><li><strong>Put a guardrail up before anyone else touches it.</strong> Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.</li></ol>



<p class="wp-block-paragraph">Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;We gzip our CSVs, so we already save the money.&#8221;</strong> Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.</li><li><strong>&#8220;Parquet is always faster.&#8221;</strong> Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.</li><li><strong>&#8220;We&#8217;ll just convert everything to Parquet.&#8221;</strong> Converting a landing zone that receives appended events all day means fighting the format&#8217;s biggest weakness. Land raw, convert on a schedule, query the converted copy.</li><li><strong>&#8220;Storage is cheap so format doesn&#8217;t matter.&#8221;</strong> Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.</li><li><strong>&#8220;Our BI tool caches, so nobody re-scans.&#8221;</strong> Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A short sequence of questions, in order.</p>



<ul class="wp-block-list"><li><strong>Is this data leaving your organisation?</strong> CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.</li><li><strong>Is it arriving continuously, one record at a time?</strong> Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.</li><li><strong>Is it read more than once, by more than one person, with filters?</strong> Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.</li><li><strong>Is the whole dataset smaller than a few hundred megabytes?</strong> Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.</li><li><strong>Are you fighting append semantics, concurrent writers, or schema evolution?</strong> You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.</li></ul>



<p class="wp-block-paragraph">The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV?</h3>



<p class="wp-block-paragraph">It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.</p>



<h3 class="wp-block-heading">Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?</h3>



<p class="wp-block-paragraph">Less than you would think, because once data is loaded into a BigQuery native table it lives in Google&#8217;s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.</p>



<h3 class="wp-block-heading">Is ORC better than Parquet?</h3>



<p class="wp-block-paragraph">They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet&#8217;s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.</p>



<h3 class="wp-block-heading">Why did my query cost the same after I converted to Parquet?</h3>



<p class="wp-block-paragraph">Usually one of three things. You are running <code>SELECT *</code>, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.</p>



<h3 class="wp-block-heading">Should I use Snappy, GZIP or ZSTD for Parquet?</h3>



<p class="wp-block-paragraph">Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.</p>



<h3 class="wp-block-heading">Can I query CSV and Parquet in the same table?</h3>



<p class="wp-block-paragraph">Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.</p>



<h3 class="wp-block-heading">What about the cost of the conversion job itself?</h3>



<p class="wp-block-paragraph">Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.</p>



<p class="wp-block-paragraph">Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help cutting a data lake bill?</h2>



<p class="wp-block-paragraph">Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves</li><li>Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses</li><li>Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control</li><li>Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges</li><li>Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types</li><li>Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end</li></ul>



<p class="wp-block-paragraph">If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>S3 Tables Compaction in Production: The Cost of Convenience</title>
		<link>https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 22 Oct 2026 06:02:31 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Optimistic Concurrency]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Query Performance]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Small Files]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<category><![CDATA[Write Amplification]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=773</guid>

					<description><![CDATA[<p>Managed Iceberg compaction in S3 Tables removes real operational work, but it will not tell you when it stops. A practical look at the maintenance jobs that fail silently, the retention cascade that keeps superseded data on your bill for two weeks, how compaction charges actually accumulate, and the handful of knobs worth touching.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The table bucket had been in production a couple of months. Ingest volume was flat, retention said ninety days, and the storage line on the bill kept climbing anyway. Compaction was enabled. The console showed nothing wrong. Someone had already checked twice.</p>



<p class="wp-block-paragraph">The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.</p>



<p class="wp-block-paragraph">That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.</p>



<h2 class="wp-block-heading">What managed compaction actually does</h2>



<p class="wp-block-paragraph">S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.</p>



<p class="wp-block-paragraph">Compaction is the Iceberg <code>rewriteDataFiles</code> routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table&#8217;s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.</p>



<p class="wp-block-paragraph">There are four strategies. <code>auto</code> is the default and picks for you: sort order defined in metadata means <code>sort</code>, otherwise <code>binpack</code>. <code>binpack</code> packs small files into bigger ones. <code>sort</code> orders rows hierarchically by your chosen columns so engines prune more files. <code>z-order</code> interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.</p>



<p class="wp-block-paragraph">Both <code>sort</code> and <code>z-order</code> need a <code>sort_order</code> in your Iceberg table properties and the <code>s3tables:GetTableData</code> permission to set. They also cost more per byte processed than <code>binpack</code>, because they do more work.</p>



<h2 class="wp-block-heading">Failure family one: maintenance that fails without telling anyone</h2>



<p class="wp-block-paragraph">This is the one worth reading the post for. Maintenance jobs have four states: <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, and <code>Not_Yet_Run</code>. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.</p>



<pre class="wp-block-code"><code>aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table</code></pre>



<p class="wp-block-paragraph">The response carries a block for each of <code>icebergCompaction</code>, <code>icebergSnapshotManagement</code>, and <code>icebergUnreferencedFileRemoval</code>, each with its own status, a <code>lastRunTimestamp</code>, and a <code>failureMessage</code>. The three are independent. Compaction reporting <code>Successful</code> tells you nothing about whether the other two are running.</p>



<p class="wp-block-paragraph">The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.</p>



<h3 class="wp-block-heading">Tags and branches disable snapshot expiry for the whole table</h3>



<p class="wp-block-paragraph">If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.</p>



<p class="wp-block-paragraph">Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month&#8217;s bill rather than in their terminal.</p>



<h3 class="wp-block-heading">Iceberg retention properties conflict with the managed config</h3>



<p class="wp-block-paragraph">If <code>history.expire.max-snapshot-age-ms</code> or <code>history.expire.min-snapshots-to-keep</code> is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in <code>metadata.json</code>. It refuses instead.</p>



<p class="wp-block-paragraph">This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.</p>



<pre class="wp-block-code"><code>ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');</code></pre>



<h3 class="wp-block-heading">gc.enabled set to false</h3>



<p class="wp-block-paragraph">Setting the Iceberg <code>gc.enabled</code> table property to <code>false</code> stops both snapshot management and unreferenced file removal. Leave it unset or explicitly <code>true</code>. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.</p>



<h3 class="wp-block-heading">KMS key policy missing the maintenance principal</h3>



<p class="wp-block-paragraph">If the table is encrypted with a customer managed key, that key&#8217;s policy has to grant access to the service principal <code>maintenance.s3tables.amazonaws.com</code>. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.</p>



<h2 class="wp-block-heading">Failure family two: the storage tail you keep paying for</h2>



<p class="wp-block-paragraph">Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.</p>



<p class="wp-block-paragraph">When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from <code>minSnapshotsToKeep</code> (default 1) and <code>maxSnapshotAgeHours</code> (default 120). Both conditions must be satisfied before a snapshot goes.</p>



<p class="wp-block-paragraph">When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for <code>nonCurrentDays</code> (default 10), then are deleted permanently.</p>



<p class="wp-block-paragraph">Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through <code>unreferencedDays</code> (default 3) into the same noncurrent window.</p>



<p class="wp-block-paragraph">Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.</p>



<pre class="wp-block-code"><code># Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'</code></pre>



<p class="wp-block-paragraph">Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.</p>



<h2 class="wp-block-heading">Failure family three: write amplification and the compaction charge</h2>



<p class="wp-block-paragraph">S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.</p>



<p class="wp-block-paragraph">The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.</p>



<p class="wp-block-paragraph">People reach for <code>targetFileSizeMB</code> first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.</p>



<p class="wp-block-paragraph">The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.</p>



<p class="wp-block-paragraph">Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.</p>



<h2 class="wp-block-heading">Failure family four: files compaction quietly skips</h2>



<p class="wp-block-paragraph">Compaction supports Parquet, Avro, and ORC. It does not support the <code>Fixed</code> data type, nor the <code>brotli</code> or <code>lz4</code> compression codecs.</p>



<p class="wp-block-paragraph">That last one matters because <code>lz4</code> is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.</p>



<p class="wp-block-paragraph">Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the <code>write.format.default</code> table property to <code>avro</code> or <code>orc</code>. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.</p>



<h2 class="wp-block-heading">Failure family five: compaction and your writers fighting over the table</h2>



<p class="wp-block-paragraph">Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.</p>



<p class="wp-block-paragraph">When compaction loses you see <code>Failed</code> with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between <code>Failed</code> and <code>Successful</code>.</p>



<p class="wp-block-paragraph">When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.</p>



<h2 class="wp-block-heading">Turning managed compaction off, and when that is right</h2>



<p class="wp-block-paragraph">Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.</p>



<p class="wp-block-paragraph">The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.</p>



<p class="wp-block-paragraph">Disabling is a per-table change, and it does not require zeroing the settings.</p>



<pre class="wp-block-code"><code>aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'</code></pre>



<p class="wp-block-paragraph">From there you run maintenance yourself. Trino exposes an <code>optimize</code> procedure through <code>ALTER TABLE ... EXECUTE</code>, and Spark has the standard Iceberg <code>rewrite_data_files</code> procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.</p>



<p class="wp-block-paragraph">Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting S3 Tables compaction and maintenance</h2>



<p class="wp-block-paragraph">Work through these in order. The first step resolves more cases than the rest combined.</p>



<ol class="wp-block-list">
<li><strong>Check all three job statuses, not one.</strong> Run <code>get-table-maintenance-job-status</code> and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.</li>

<li><strong>Read the failure message literally.</strong> It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means <code>history.expire.*</code> is set.</li>

<li><strong>Status is <code>Disabled</code> or <code>Not_Yet_Run</code> unexpectedly.</strong> Someone replaced the configuration. Confirm with <code>get-table-maintenance-configuration</code>, and remember <code>put-table-maintenance-configuration</code> replaces rather than merges.</li>

<li><strong>Compaction runs but file count stays high.</strong> Check codec and data types against the unsupported list. Files using <code>brotli</code> or <code>lz4</code>, or containing the <code>Fixed</code> type, are outside what compaction touches.</li>

<li><strong>Insufficient-access failures on an encrypted table.</strong> Check the KMS key policy grants the <code>maintenance.s3tables.amazonaws.com</code> service principal, and that the key is enabled and still exists.</li>

<li><strong>Storage still growing with everything green.</strong> Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.</li>
</ol>



<p class="wp-block-paragraph">For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer &#8220;when did this start&#8221; rather than &#8220;what is broken now&#8221;.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating the three maintenance jobs as one thing.</strong> They fail independently and for different reasons. &#8220;Maintenance is enabled&#8221; is not a status.</li>

<li><strong>Creating branches or tags on a managed table.</strong> A normal Iceberg workflow that switches off snapshot expiry for the whole table.</li>

<li><strong>Tuning <code>targetFileSizeMB</code> to reduce cost.</strong> It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.</li>

<li><strong>Setting <code>sort</code> or <code>z-order</code> without a sort order defined.</strong> The strategy needs <code>sort_order</code> in table properties and the <code>s3tables:GetTableData</code> permission, and it costs more per byte than <code>binpack</code>.</li>

<li><strong>Running write pipelines with no commit retry.</strong> Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.</li>
</ul>



<h2 class="wp-block-heading">Best practices for running S3 Tables compaction in production</h2>



<ul class="wp-block-list">
<li><strong>Poll job status on a schedule and alert on it.</strong> A small scheduled function calling <code>get-table-maintenance-job-status</code> across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on <code>Failed</code> and on a stale <code>lastRunTimestamp</code>, on whatever dashboard you already watch, Grafana or CloudWatch.</li>

<li><strong>Make maintenance config part of your IaC.</strong> These are settings someone can change with one CLI call and no review. Codify them so drift is visible.</li>

<li><strong>Set retention windows from a real requirement.</strong> Ask how far back anyone needs to time travel, then set <code>maxSnapshotAgeHours</code> and <code>nonCurrentDays</code> to that plus a margin, rather than accepting defaults by omission.</li>

<li><strong>Batch on the write side before tuning anything else.</strong> Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.</li>

<li><strong>Choose the strategy from the query pattern.</strong> One dominant filter column suggests <code>sort</code>. Several equally weighted predicates suggest <code>z-order</code>. Neither suggests leaving it on <code>auto</code>.</li>

<li><strong>Track the compaction billing dimension separately.</strong> It moves independently of storage and it is the earliest signal that a writer&#8217;s behaviour has changed.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I know if S3 Tables compaction is actually running?</h3>



<p class="wp-block-paragraph">Call <code>get-table-maintenance-job-status</code> and read the <code>icebergCompaction</code> block. It returns <code>Successful</code>, <code>Failed</code>, <code>Disabled</code>, or <code>Not_Yet_Run</code>, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.</p>



<h3 class="wp-block-heading">Why is my S3 Tables storage growing when compaction says it succeeded?</h3>



<p class="wp-block-paragraph">Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.</p>



<h3 class="wp-block-heading">Can I run my own compaction alongside the managed one?</h3>



<p class="wp-block-paragraph">You can, but it is a poor idea. Both are writers competing under Iceberg&#8217;s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.</p>



<h3 class="wp-block-heading">Does disabling compaction stop all S3 Tables charges?</h3>



<p class="wp-block-paragraph">No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.</p>



<h3 class="wp-block-heading">Is sort or z-order compaction worth the extra cost?</h3>



<p class="wp-block-paragraph">It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour <code>sort</code>. Several dimensions in varying combinations favour <code>z-order</code>. Queries that mostly scan everything justify neither.</p>



<h3 class="wp-block-heading">Can I recover data deleted by S3 Tables maintenance?</h3>



<p class="wp-block-paragraph">Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.</p>



<p class="wp-block-paragraph">What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on <code>get-table-maintenance-job-status</code> that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.</p>



<h2 class="wp-block-heading">Need help with Iceberg tables on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots</li>

<li>Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana</li>

<li>Working out where a bucket&#8217;s storage growth actually comes from, and setting retention windows against a real requirement</li>

<li>Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them</li>

<li>Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick</li>

<li>Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance</li>
</ul>



<p class="wp-block-paragraph">If something is not adding up, send me the output of <code>get-table-maintenance-job-status</code> for the table and a rough description of your write pattern. That is usually enough to say what is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/s3-tables-compaction-production/">S3 Tables Compaction in Production: The Cost of Convenience</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</title>
		<link>https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 06:06:48 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=500</guid>

					<description><![CDATA[<p>S3 holds the documents, Iceberg holds the rows, and everything works right up until a number disagrees with its source PDF. Combining structured and unstructured data on AWS is an identity problem, not a storage one. Here are the five failure families that cause it, why each stays invisible until it isn't, and the ingestion order that avoids most of them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;the contract value in the dashboard is wrong.&#8221; It wasn&#8217;t. The number in the warehouse table was right for the version of the PDF that had been extracted eleven months earlier. Someone had uploaded an amended contract to the same S3 prefix, the extraction job had picked it up, written a new row, and nothing downstream knew which of the two rows was current. Both were true. Only one was correct.</p>



<p class="wp-block-paragraph">That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.</p>



<p class="wp-block-paragraph">The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.</p>



<h2 class="wp-block-heading">What this post covers</h2>



<p class="wp-block-paragraph">Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn&#8217;t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.</p>



<p class="wp-block-paragraph">Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: there is no join key, only a filename</h2>



<p class="wp-block-paragraph">Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.</p>



<p class="wp-block-paragraph">Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.</p>



<h3 class="wp-block-heading">What to use instead</h3>



<p class="wp-block-paragraph">Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.</p>



<p class="wp-block-paragraph">Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for &#8220;this exact bytes-on-disk artefact.&#8221; If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.</p>



<p class="wp-block-paragraph">S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.</p>



<h2 class="wp-block-heading">Failure family two: extraction confidence gets flattened into a fact</h2>



<p class="wp-block-paragraph">Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.</p>



<p class="wp-block-paragraph">Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn&#8217;t go away. You just stopped being able to see it.</p>



<p class="wp-block-paragraph">Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always &#8220;which extractor produced these,&#8221; and without that column the answer is a week of archaeology.</p>



<pre class="wp-block-code"><code>-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);</code></pre>



<p class="wp-block-paragraph">The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.</p>



<h2 class="wp-block-heading">Failure family three: two catalogs, two permission models</h2>



<p class="wp-block-paragraph">This is the one that bites teams who did everything else right.</p>



<p class="wp-block-paragraph">Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.</p>



<p class="wp-block-paragraph">The result is predictable and unpleasant. An analyst who is correctly denied the <code>salary</code> column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.</p>



<p class="wp-block-paragraph">There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:</p>



<ul class="wp-block-list">

<li>Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.</li>


<li>Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.</li>


<li>Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.</li>


<li>Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.</li>

</ul>



<p class="wp-block-paragraph">Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with <code>.metadata.json</code> appended. That file is where classification, tenant, document ID and effective dates belong.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}</code></pre>



<p class="wp-block-paragraph">Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.</p>



<h2 class="wp-block-heading">Failure family four: index decisions you cannot take back</h2>



<p class="wp-block-paragraph">S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.</p>



<p class="wp-block-paragraph">By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.</p>



<p class="wp-block-paragraph">If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.</p>



<pre class="wp-block-code"><code># Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'</code></pre>



<p class="wp-block-paragraph">The dimension has to match your embedding model&#8217;s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.</p>



<p class="wp-block-paragraph">Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing rewrites history</h2>



<p class="wp-block-paragraph">You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter&#8217;s board pack change.</p>



<p class="wp-block-paragraph">This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.</p>



<p class="wp-block-paragraph">Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and &#8220;why did this number move&#8221; has an answer you can query instead of guess at.</p>



<p class="wp-block-paragraph">Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.</p>



<h2 class="wp-block-heading">A shape that works for combining structured and unstructured data on AWS</h2>



<p class="wp-block-paragraph">Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.</p>



<ol class="wp-block-list">

<li><strong>Land raw, enable versioning, enable S3 Metadata.</strong> Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.</li>


<li><strong>Mint the document ID at landing.</strong> Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.</li>


<li><strong>Classify before you extract.</strong> Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.</li>


<li><strong>Extract to a staging table, not straight to the curated table.</strong> Staging gives you somewhere to run validation before anything touches a table an analyst can query.</li>


<li><strong>Merge into Iceberg on document identity, carrying confidence and provenance columns.</strong> Never merge on object key.</li>


<li><strong>Embed separately, keyed by the same document ID.</strong> The vector plane and the table plane are different systems with different refresh rates. Treat them as such.</li>


<li><strong>Join at query time in Athena.</strong> Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.</li>

</ol>



<p class="wp-block-paragraph">Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Athena says the principal has no privilege on the resource</h3>



<p class="wp-block-paragraph">This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.</p>



<h3 class="wp-block-heading">Queries against metadata tables fail with a syntax error</h3>



<p class="wp-block-paragraph">Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.</p>



<h3 class="wp-block-heading">Knowledge base sync fails partway with validation errors</h3>



<p class="wp-block-paragraph">Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.</p>



<h3 class="wp-block-heading">Retrieval returns documents the caller should not see</h3>



<p class="wp-block-paragraph">Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.</p>



<h3 class="wp-block-heading">Athena gets slower week over week on a table that isn&#8217;t growing much</h3>



<p class="wp-block-paragraph">Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Joining on the S3 object key because it was there and it worked in the prototype.</li>


<li>Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.</li>


<li>Applying a confidence threshold and discarding the score.</li>


<li>Creating the vector index before deciding on the embedding model.</li>


<li>Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.</li>


<li>Reprocessing in place, with no record of which extractor produced which value.</li>


<li>Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.</li>


<li>Confidence and provenance columns are not optional. They are the difference between debugging and guessing.</li>


<li>Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.</li>


<li>Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.</li>


<li>Enforce classification in all three planes independently. There is no single choke point.</li>


<li>Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.</li>


<li>Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a lakehouse to combine structured and unstructured data on AWS?</h3>



<p class="wp-block-paragraph">No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.</p>



<h3 class="wp-block-heading">Should extraction output live in the same table as the structured source data?</h3>



<p class="wp-block-paragraph">Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.</p>



<h3 class="wp-block-heading">Is S3 Vectors a replacement for OpenSearch?</h3>



<p class="wp-block-paragraph">For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.</p>



<h3 class="wp-block-heading">How do I keep the vector index in sync with the documents?</h3>



<p class="wp-block-paragraph">Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because &#8220;mostly in sync&#8221; degrades quietly.</p>



<h3 class="wp-block-heading">What breaks first when this architecture is done badly?</h3>



<p class="wp-block-paragraph">Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.</p>



<h3 class="wp-block-heading">Can I use Textract instead of Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.</p>



<h3 class="wp-block-heading">Does Lake Formation govern my documents as well as my tables?</h3>



<p class="wp-block-paragraph">It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.</p>



<p class="wp-block-paragraph">Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help wiring documents and tables together?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:</p>



<ul class="wp-block-list">

<li>Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.</li>


<li>Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.</li>


<li>Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.</li>


<li>Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.</li>


<li>Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.</li>


<li>Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.</li>

</ul>



<p class="wp-block-paragraph">Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
