<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>SageMaker Unified Studio | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/sagemaker-unified-studio/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/sagemaker-unified-studio/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 27 Oct 2026 06:05:56 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>SageMaker Unified Studio | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/sagemaker-unified-studio/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</title>
		<link>https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 27 Oct 2026 06:05:56 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon DataZone]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudFormation]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[IAM Identity Center]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[SageMaker AI]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=513</guid>

					<description><![CDATA[<p>SageMaker Unified Studio is a front door, not a runtime. It consolidates the interface while leaving your bill, your IAM design and your infrastructure as code exactly where they were, and it quietly hands you a DataZone governance system to operate. Here is what it genuinely replaces, what it doesn't, the failure modes that bite during adoption, and how to decide whether it is worth it for your team.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question always arrives in the same shape. A team stands up a domain, everyone logs in, the demo goes well, and then somebody asks when they can turn off the Glue console and the Athena query editor.</p>



<p class="wp-block-paragraph">They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.</p>



<p class="wp-block-paragraph">This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.</p>



<h2 class="wp-block-heading">SageMaker Unified Studio is a front door, not a runtime</h2>



<p class="wp-block-paragraph">Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.</p>



<p class="wp-block-paragraph">AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project&#8217;s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new <em>authorisation</em> layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.</p>



<h2 class="wp-block-heading">What it actually replaces</h2>



<h3 class="wp-block-heading">The pile of standalone studios and query editors</h3>



<p class="wp-block-paragraph">This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.</p>



<p class="wp-block-paragraph">The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a &#8220;workspace&#8221; meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.</p>



<h3 class="wp-block-heading">Per-team domain sprawl</h3>



<p class="wp-block-paragraph">The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.</p>



<h3 class="wp-block-heading">The access request spreadsheet</h3>



<p class="wp-block-paragraph">There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.</p>



<p class="wp-block-paragraph">The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.</p>



<h2 class="wp-block-heading">What it doesn&#8217;t replace</h2>



<h3 class="wp-block-heading">Your bill, or your ability to explain it</h3>



<p class="wp-block-paragraph">There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.</p>



<p class="wp-block-paragraph">What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer&#8217;s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.</p>



<p class="wp-block-paragraph">Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.</p>



<h3 class="wp-block-heading">Your IAM and Lake Formation design</h3>



<p class="wp-block-paragraph">Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.</p>



<p class="wp-block-paragraph">Nobody in the adoption meeting will say &#8220;this fixes our permissions mess.&#8221; Plenty of people will quietly assume it. Say it out loud early.</p>



<h3 class="wp-block-heading">Your infrastructure as code</h3>



<p class="wp-block-paragraph">This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.</p>



<p class="wp-block-paragraph">It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.</p>



<h3 class="wp-block-heading">SageMaker AI</h3>



<p class="wp-block-paragraph">The original SageMaker was renamed SageMaker AI and remains available on its own. If your team&#8217;s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.</p>



<h2 class="wp-block-heading">What SageMaker Unified Studio quietly adds</h2>



<p class="wp-block-paragraph">Here is the section that rarely makes it into the migration plan.</p>



<h3 class="wp-block-heading">You now operate DataZone, intentionally or not</h3>



<p class="wp-block-paragraph">SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a <code>dzd</code> prefix, and the console for creating one still lives under the DataZone console path.</p>



<p class="wp-block-paragraph">You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:</p>



<pre class="wp-block-code"><code># Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID</code></pre>



<p class="wp-block-paragraph">Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.</p>



<h3 class="wp-block-heading">Project creation is a CloudFormation deploy wearing a friendly hat</h3>



<p class="wp-block-paragraph">Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.</p>



<p class="wp-block-paragraph">This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation &#8220;just doesn&#8217;t work,&#8221; do not reproduce it in the UI. Read the stack events:</p>



<pre class="wp-block-code"><code># Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table</code></pre>



<p class="wp-block-paragraph">The first failed resource causes the rollback. Everything after it is noise. This one habit turns most &#8220;the platform is broken&#8221; tickets into five-minute answers.</p>



<h3 class="wp-block-heading">A domain-type fork you will live with for a long time</h3>



<p class="wp-block-paragraph">Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.</p>



<p class="wp-block-paragraph">Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post&#8217;s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.</p>



<h2 class="wp-block-heading">Where adoption actually goes wrong</h2>



<h3 class="wp-block-heading">&#8220;It works in my console but not in the project&#8221;</h3>



<p class="wp-block-paragraph">This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.</p>



<p class="wp-block-paragraph">Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:</p>



<pre class="wp-block-code"><code>aws sts get-caller-identity</code></pre>



<p class="wp-block-paragraph">If the ARN is not the one you assumed, stop. You have found the problem.</p>



<h3 class="wp-block-heading">The bill went up after a migration that was supposed to save money</h3>



<p class="wp-block-paragraph">Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.</p>



<h3 class="wp-block-heading">Nobody can find the data that is definitely there</h3>



<p class="wp-block-paragraph">Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Treating it as a migration rather than an addition.</strong> Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.</li>
<li><strong>Letting users self-serve projects on day one.</strong> Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.</li>
<li><strong>Choosing the domain authentication type in a hurry.</strong> It is the hardest thing here to change later.</li>
<li><strong>Skipping the catalog.</strong> Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.</li>
<li><strong>Assuming the CI/CD CLI provisions your platform.</strong> It deploys applications into projects that already exist.</li>
<li><strong>Not tagging project resources from the start.</strong> Retrofitting attribution across dozens of blueprint-created resources is miserable work.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">A rough order of operations for anyone weighing this up:</p>



<ol class="wp-block-list">
<li><strong>Ask what problem you are solving.</strong> If it is &#8220;our teams cannot discover or safely share each other&#8217;s data,&#8221; this is the right tool. If it is &#8220;we have too many consoles,&#8221; it is an expensive way to buy fewer bookmarks.</li>
<li><strong>Count your teams.</strong> The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.</li>
<li><strong>Decide the authentication model first.</strong> Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.</li>
<li><strong>Write the project profiles yourself.</strong> The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.</li>
<li><strong>Prove the teardown path before the build path.</strong> Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.</li>
<li><strong>Keep the old consoles.</strong> They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.</li>
</ol>



<p class="wp-block-paragraph">One more piece of judgement: wire your existing observability in from the start rather than relying on the studio&#8217;s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Unified Studio replace SageMaker Studio?</h3>



<p class="wp-block-paragraph">It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.</p>



<h3 class="wp-block-heading">Do I still need Amazon DataZone separately?</h3>



<p class="wp-block-paragraph">Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.</p>



<h3 class="wp-block-heading">How much does SageMaker Unified Studio cost?</h3>



<p class="wp-block-paragraph">The interface has no standalone charge. You pay for each AWS service you use through it on that service&#8217;s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.</p>



<h3 class="wp-block-heading">Can I manage domains and projects with Terraform?</h3>



<p class="wp-block-paragraph">AWS&#8217;s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.</p>



<h3 class="wp-block-heading">What happens to my Glue jobs and Athena queries?</h3>



<p class="wp-block-paragraph">Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.</p>



<h3 class="wp-block-heading">Is it worth adopting for a small team?</h3>



<p class="wp-block-paragraph">Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.</p>



<h3 class="wp-block-heading">Can I run both the old and new experiences at once?</h3>



<p class="wp-block-paragraph">Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.</p>



<p class="wp-block-paragraph">That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was &#8220;fewer tabs.&#8221; The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.</p>



<p class="wp-block-paragraph">Ask which one you are before the first project gets created, not after the fortieth.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your adoption plan?</h2>



<p class="wp-block-paragraph">This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup</li>
<li>Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults</li>
<li>Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use</li>
<li>Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover</li>
<li>Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants</li>
<li>Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long</li>
</ul>



<p class="wp-block-paragraph">Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sagemaker-unified-studio-what-it-replaces/">SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
