They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.

This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.

SageMaker Unified Studio is a front door, not a runtime

Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.

AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project’s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new authorisation layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.

What it actually replaces

The pile of standalone studios and query editors

This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.

The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a “workspace” meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.

Per-team domain sprawl

The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.

The access request spreadsheet

There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.

The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.

What it doesn’t replace

Your bill, or your ability to explain it

There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.

What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer’s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.

Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.

Your IAM and Lake Formation design

Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.

Nobody in the adoption meeting will say “this fixes our permissions mess.” Plenty of people will quietly assume it. Say it out loud early.

Your infrastructure as code

This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.

It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.

SageMaker AI

The original SageMaker was renamed SageMaker AI and remains available on its own. If your team’s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.

What SageMaker Unified Studio quietly adds

Here is the section that rarely makes it into the migration plan.

You now operate DataZone, intentionally or not

SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a dzd prefix, and the console for creating one still lives under the DataZone console path.

You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:

# Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID

Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.

Project creation is a CloudFormation deploy wearing a friendly hat

Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.

This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation “just doesn’t work,” do not reproduce it in the UI. Read the stack events:

# Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table

The first failed resource causes the rollback. Everything after it is noise. This one habit turns most “the platform is broken” tickets into five-minute answers.

A domain-type fork you will live with for a long time

Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.

Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post’s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.

Where adoption actually goes wrong

“It works in my console but not in the project”

This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.

Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:

aws sts get-caller-identity

If the ARN is not the one you assumed, stop. You have found the problem.

The bill went up after a migration that was supposed to save money

Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.

Nobody can find the data that is definitely there

Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.

Common mistakes

• Treating it as a migration rather than an addition. Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.
• Letting users self-serve projects on day one. Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.
• Choosing the domain authentication type in a hurry. It is the hardest thing here to change later.
• Skipping the catalog. Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.
• Assuming the CI/CD CLI provisions your platform. It deploys applications into projects that already exist.
• Not tagging project resources from the start. Retrofitting attribution across dozens of blueprint-created resources is miserable work.

How I’d decide

A rough order of operations for anyone weighing this up:

1. Ask what problem you are solving. If it is “our teams cannot discover or safely share each other’s data,” this is the right tool. If it is “we have too many consoles,” it is an expensive way to buy fewer bookmarks.
2. Count your teams. The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.
3. Decide the authentication model first. Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.
4. Write the project profiles yourself. The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.
5. Prove the teardown path before the build path. Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.
6. Keep the old consoles. They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.

One more piece of judgement: wire your existing observability in from the start rather than relying on the studio’s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.

Frequently asked questions

Does SageMaker Unified Studio replace SageMaker Studio?

It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.

Do I still need Amazon DataZone separately?

Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.

How much does SageMaker Unified Studio cost?

The interface has no standalone charge. You pay for each AWS service you use through it on that service’s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.

Can I manage domains and projects with Terraform?

AWS’s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.

What happens to my Glue jobs and Athena queries?

Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.

Is it worth adopting for a small team?

Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.

Can I run both the old and new experiences at once?

Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.

The one thing worth remembering

SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.

That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was “fewer tabs.” The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.

Ask which one you are before the first project gets created, not after the fortieth.

Need a second opinion on your adoption plan?

This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:

• Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup
• Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults
• Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use
• Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover
• Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants
• Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long

Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.

The post SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t appeared first on John Nessime.

That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.

That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.

The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.

What this post covers

Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn’t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.

Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.

Failure family one: there is no join key, only a filename

Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.

Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.

What to use instead

Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.

Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for “this exact bytes-on-disk artefact.” If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.

S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.

Failure family two: extraction confidence gets flattened into a fact

Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.

Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn’t go away. You just stopped being able to see it.

Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always “which extractor produced these,” and without that column the answer is a week of archaeology.

-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);

The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.

Failure family three: two catalogs, two permission models

This is the one that bites teams who did everything else right.

Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.

The result is predictable and unpleasant. An analyst who is correctly denied the salary column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.

There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:

• Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.
• Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.
• Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.
• Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.

Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with .metadata.json appended. That file is where classification, tenant, document ID and effective dates belong.

{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}

Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.

Failure family four: index decisions you cannot take back

S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.

By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.

If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.

# Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'

The dimension has to match your embedding model’s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.

Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.

Failure family five: reprocessing rewrites history

You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter’s board pack change.

This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.

Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and “why did this number move” has an answer you can query instead of guess at.

Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.

A shape that works for combining structured and unstructured data on AWS

Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.

1. Land raw, enable versioning, enable S3 Metadata. Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.
2. Mint the document ID at landing. Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.
3. Classify before you extract. Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.
4. Extract to a staging table, not straight to the curated table. Staging gives you somewhere to run validation before anything touches a table an analyst can query.
5. Merge into Iceberg on document identity, carrying confidence and provenance columns. Never merge on object key.
6. Embed separately, keyed by the same document ID. The vector plane and the table plane are different systems with different refresh rates. Treat them as such.
7. Join at query time in Athena. Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.

Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.

Troubleshooting

Athena says the principal has no privilege on the resource

This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.

Queries against metadata tables fail with a syntax error

Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.

Knowledge base sync fails partway with validation errors

Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.

Retrieval returns documents the caller should not see

Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.

Athena gets slower week over week on a table that isn’t growing much

Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.

Common mistakes

• Joining on the S3 object key because it was there and it worked in the prototype.
• Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.
• Applying a confidence threshold and discarding the score.
• Creating the vector index before deciding on the embedding model.
• Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.
• Reprocessing in place, with no record of which extractor produced which value.
• Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.

Best practices

• One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.
• Confidence and provenance columns are not optional. They are the difference between debugging and guessing.
• Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.
• Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.
• Enforce classification in all three planes independently. There is no single choke point.
• Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.
• Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.

FAQ

Do I need a lakehouse to combine structured and unstructured data on AWS?

No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.

Should extraction output live in the same table as the structured source data?

Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.

Is S3 Vectors a replacement for OpenSearch?

For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.

How do I keep the vector index in sync with the documents?

Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because “mostly in sync” degrades quietly.

What breaks first when this architecture is done badly?

Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.

Can I use Textract instead of Bedrock Data Automation?

Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.

Does Lake Formation govern my documents as well as my tables?

It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.

The one thing worth remembering

Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.

Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.

Need help wiring documents and tables together?

This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:

• Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.
• Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.
• Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.
• Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.
• Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.
• Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.

Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.

The post The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS appeared first on John Nessime.

That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.

This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.

First, understand what you are actually migrating

The framing “SageMaker Lakehouse versus Glue and Athena” is misleading, and it drives bad decisions.

SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.

The unit it introduces is the catalog: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A managed catalog holds data you own in S3 or Redshift Managed Storage. A federated catalog points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.

Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.

So the question is not “should we move our data”. It is “should we move our governance”.

The failure mode that actually bites

Lake Formation grants Super permission to a virtual group called IAMAllowedPrincipals on every existing Data Catalog database and table by default. That group effectively means “anyone whose IAM policy already lets them in”. It exists purely for backward compatibility. While it holds Super on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.

So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke IAMAllowedPrincipals so the grants take effect.

That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.

It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.

Hybrid access mode is the way through

Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.

The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.

The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.

Where the lakehouse genuinely wins

• One permission surface across engines. If governance today is “Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB”, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.
• Tag-based access control across federated sources. LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.
• Querying across boundaries without copying. A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.
• The Iceberg REST interface. Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.
• Cross-account sharing without replication. For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.

Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.

Where Glue plus Athena is still the right answer

• Single account, S3 only, one team. Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.
• Your access control is coarse and that is correct. Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.
• You have heavy IAM-only automation. Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.
• Your bottleneck is query cost, not governance. Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.
• You depend on a specific engine behaviour today. Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.

The cost levers that move, and the ones that don’t

Pricing pages change, so here are the billing mechanisms rather than numbers.

Unchanged: metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.

New line items:

• Lambda invocations behind federated catalogs. When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.
• Service-managed Redshift Serverless compute. Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.
• Redshift Managed Storage. Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.

The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:

# Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -> Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"

Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog’s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.

Is a SageMaker Lakehouse migration worth it? A decision procedure

Run these in order. Stop at the first one that gives a clear answer.

1. Count your non-S3 governed sources. Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.
2. Inventory every principal touching the Data Catalog. Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.
3. Ask whether column or row-level control has actually been requested. If the answer is a compliance requirement with a name attached, proceed. If it is “it would be nice”, you are buying a permission model to solve a hypothetical.
4. Check engine coverage against your real workloads. Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.
5. Price the federated path against a copy. For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.
6. Decide the destination, not just the direction. “Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets” is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.

Three things to run before you commit

Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:

aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }

S3 Tables catalogs use a s3tablescatalog/bucket-name naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on list-data-catalogs will quietly under-report, so enumerate S3 Tables catalogs separately.

Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:

aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'

This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.

Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:

--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true

Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.

Arguments that don’t survive contact

• “It’s the same Glue Data Catalog, so migrating is low risk.” True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.
• “We’ll migrate to avoid falling behind.” AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.
• “Federated catalogs mean we can retire our ETL.” Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.
• “Iceberg REST support means no lock-in.” The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.
• “We can test this in a sandbox.” You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.

How I’d decide

S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.

Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a “clean” fully-enforced state is aesthetic, not operational.

Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.

Frequently asked questions

Does SageMaker Lakehouse replace AWS Glue and Athena?

No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.

Will a SageMaker Lakehouse migration break my existing Athena queries?

Reorganising into catalogs does not break queries on its own. What breaks them is revoking Super from IAMAllowedPrincipals before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.

Do I have to move data into Redshift Managed Storage?

No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.

How much extra does it cost?

The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.

Is SageMaker Unified Studio required?

No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.

What is the smallest useful first step?

Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.

Conclusion

A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.

That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.

Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke IAMAllowedPrincipals until you can name every principal that will notice.

Need help deciding, or unpicking a migration already underway?

An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:

• Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked
• Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage
• Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for
• Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants
• Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later
• Honest second opinions on whether a migration is worth doing at all, including the answer that it isn’t

Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I’ll tell you what I’d do with it.

The post SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It? appeared first on John Nessime.

That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between “we scanned this” and “we found nothing” is where most Macie deployments quietly fail their first audit.

This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.

The eligibility chain behind Amazon Macie PII detection

Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.

1. Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.
2. Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.
3. Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.
4. Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?
5. Does the content parse, and stay inside the per-file quotas?
6. Do the active data identifiers match anything in it?

Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.

One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label Not yet analyzed. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.

Gap one: objects Macie never opened

Macie calls these unclassifiable. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I’ve seen, and the cheapest to fix.

Missing file extensions

Classifiability comes from the file name extension. A file full of customer records named part-00003-a4f9 is invisible. The same bytes named part-00003-a4f9.json get inspected and produce findings.

This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.

Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can’t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.

Unsupported storage classes

A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.

Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:

# Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total > `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table

Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket’s sensitivity score means nothing.

Gap two: objects it opened and couldn’t read

These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.

• Customer-provided keys (SSE-C). Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.
• KMS key policy. For customer managed keys, the key policy must allow the Macie service-linked role, AWSServiceRoleForAmazonMacie, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the aws-samples/amazon-macie-scripts repository that enumerates every key Macie needs and generates the CLI commands to fix them.
• Restrictive bucket policies. An explicit Deny conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the aws:PrincipalArn condition key.

To see which objects Macie actually touched in a bucket, pull the object samples. It’s the closest thing to a scan log you get:

# Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name

Anything with a status of SKIPPED is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.

Gap three: it read the file and nothing matched

Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.

Add the managed identifiers for your actual jurisdictions. If you hold records for customers in a country whose national ID identifier isn’t in the recommended set, you’re scanning for the wrong things with perfect efficiency. Pull the current list with aws macie2 list-managed-data-identifiers and compare it against where your customers live.

Write custom identifiers for what only you know is sensitive. Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare [0-9]{8} matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.

Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:

# Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'

Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven’t tested.

The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone’s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.

Gap four: quota truncation inside large files

Subtle, because the object is analyzed. Just not completely, and a partial result reports like a full one.

• Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.
• Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.
• JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.
• Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.

Findings carry a status reason naming the limit that was hit, with values like FILE_EXCEEDS_SIZE_LIMIT, ARCHIVE_NESTING_LEVEL_OVER_LIMIT and JSON_NESTING_LEVEL_OVER_LIMIT. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.

Automated discovery or a discovery job?

Automated sensitive data discovery evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.

Sensitive data discovery jobs are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.

# One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'

One detail about --sampling-percentage that trips people up: it selects a random percentage of eligible objects and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.

What actually drives the bill

Macie bills on three dimensions, and the one that surprises people is not the one they budget for.

• Bucket evaluation. Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.
• Object monitoring. Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.
• Data inspected. Per GB actually analyzed, covering both automated discovery and jobs.

So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.

The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.

Getting findings somewhere a human sees them

Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.

Publish findings outward. Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that’s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.

Configure the discovery results repository. Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and “we scanned it and found nothing” is exactly the record an auditor asks for.

One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it’s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that’s a wipe tool from a vendor like O&O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.

Troubleshooting a bucket that reports nothing

When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.

1. Check the sensitivity label. Not yet analyzed at score 50 means nothing was analyzed. Do not read that as clean.
2. Check the coverage page. Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.
3. Compare classifiable and unclassifiable counts with the describe-buckets query above.
4. List a few object keys. Run aws s3 ls against a prefix and look for extensions. Extensionless keys explain the whole thing.
5. Check the storage class. A lifecycle rule may have moved everything somewhere Macie doesn’t read.
6. Check encryption. SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.
7. Check object samples with list-resource-profile-artifacts to see exactly what was selected and skipped.
8. Only now question the identifiers. Run a targeted job with a custom identifier for the data you expect and see whether it fires.

Common mistakes

• Treating a low sensitivity score as evidence of no PII. It reflects what was found and how much was analyzed, so low coverage produces a low score.
• Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.
• Running default identifiers in a business operating outside the jurisdictions the recommended set covers.
• Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.
• Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.
• Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.

Best practices

• Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.
• Fix extensions at the producer rather than building a rename pipeline downstream.
• Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.
• Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.
• Test every custom identifier against both a positive and a negative sample before it reaches a job.
• Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.
• Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.

Frequently asked questions

Can Amazon Macie scan anything other than S3?

No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.

Does Macie scan new objects as they land?

Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.

Why does a bucket I know has PII show a low sensitivity score?

Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads Not yet analyzed. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.

Do I need both automated discovery and discovery jobs?

For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.

How do I cut false positives without missing real PII?

In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.

Does Macie work across accounts in an organization?

Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator’s automated discovery settings apply to member accounts, members can’t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.

The one thing worth remembering

Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.

Do that once and the rest of the service becomes straightforward. Skip it and you’re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.

Need help getting Macie to actually see your data?

Most of the work in a Macie rollout isn’t turning it on. It’s the unglamorous part: proving what got scanned, fixing the reasons things didn’t, and making the output land somewhere a human acts on it. That’s the part I help with.

• Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap
• Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects
• Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first
• Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible
• Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise
• Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what

If you want a second opinion, send me the output of the describe-buckets query above, or a screenshot of your coverage page, and I’ll tell you what’s actually being scanned.

The post Amazon Macie PII Detection: The Buckets It Never Opened appeared first on John Nessime.

It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.

That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is “anyone the other account decided to let in.”

This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.

AWS cross-account data sharing is a delegation, not a permission

Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.

That is why “I added them to the bucket policy and it still fails” is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.

Now the part that bites. Look at how most cross-account bucket policies get written:

{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}

Despite how it reads, that :root suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.

Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:

{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}

Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.

Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, aws:PrincipalOrgID on its own is a fairly wide door.

The grant that stops working after a rebuild

When you name a role ARN as a principal, IAM stores the role’s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque AROA... string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.

A terraform destroy followed by terraform apply on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.

The encryption boundary nobody tests until it is live

Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.

• Downloading an object encrypted with SSE-KMS requires kms:Decrypt on the key.
• Uploading one requires kms:GenerateDataKey.
• A multipart upload requires both.

The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:

{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}

The kms:ViaService condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer’s own IAM policy has to allow the KMS actions too.

Now the sharp edge, which is worth reading twice. If a KMS key is specified by alias rather than by fully qualified ARN, the alias is resolved in the requester’s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.

Who owns the bytes after the handoff

Third family: ownership, and it depends entirely on which direction the data moves.

New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.

Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.

Check and fix it with the ownership controls API:

aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'

Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with AccessControlListNotSupported. Requests that send no ACL, or send bucket-owner-full-control, keep working. So the sequence is:

1. Find every writer that sets an ACL, including old scripts using --acl and any SDK call passing an ACL argument.
2. Update those clients so they stop sending one.
3. Confirm from CloudTrail that ACL-bearing PUTs have stopped.
4. Flip the bucket to Bucket owner enforced.

Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team’s control. Push scatters them.

Copies you cannot recall

Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.

Three reference-shaped patterns are worth knowing before you reach for a bulk export.

S3 Access Points for object-level sharing

Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the s3:DataAccessPointAccount condition. Access points can also be owned by the consumer’s account, so they manage their own detail without touching your bucket policy.

The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.

Lake Formation for table-level sharing

When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.

aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE

After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a resource link, which behaves roughly like a symlink in the recipient’s Data Catalog. Missing resource links account for most of the “the share went through but I cannot see the table” messages.

One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:

aws lakeformation get-data-lake-settings

When a copy really is the answer

Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst’s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.

Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&O SafeErase reads a great deal better in an audit than “we deleted the folder.” And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.

The guardrail that catches the mistake you will make

Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.

Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is “no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.” That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.

Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner’s behavior once data has legitimately left. An RCP is a wall around your side of the fence.

Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.

For sharing with a third party rather than an internal account, the external ID is not optional in my book:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}

The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.

Verifying that the grant is what you think it is

Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question “what is currently reachable from outside my zone of trust” using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.

aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access

Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.

The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.

Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns “is this still needed” from an argument into a query.

Troubleshooting the denials that are hard to read

• You can list the bucket but every GET returns 403. That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check kms:Decrypt on both the key policy and the consumer’s identity policy.
• 403 through an access point while the bucket policy is clearly correct. The access point has its own policy and it also has to allow the calling principal. Read that one next.
• AssumeRole is denied although the trust policy names the caller. Cross-account needs both halves: the caller’s identity policy must permit sts:AssumeRole against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.
• A principal in the policy displays as AROA followed by random characters. The referenced role was deleted. Recreating it under the same name will not restore access.
• A Lake Formation share appears in the catalog but queries fail. Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.
• Access worked yesterday and stopped today with nothing changed locally. Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.

Common mistakes

• Granting to an account principal when a role ARN would have done, then never revisiting it.
• Treating aws:PrincipalOrgID as least privilege. It narrows the organization, not the account.
• Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.
• Referring to KMS keys by alias in cross-account configuration instead of by full ARN.
• Letting a partner push into your bucket without confirming the ownership setting first.
• Exporting a full table when the consumer needed four columns, because the export was easier to write.
• Creating a vendor role with no external ID and no conditions because the vendor’s setup guide said to.
• Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.

Best practices worth the effort

• Give every cross-account grant a descriptive Sid. It is the only free documentation slot IAM offers, and future-you will read it.
• Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.
• Pair a specific principal with a broad condition, rather than relying on either alone.
• Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.
• Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.
• Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.
• Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.

Frequently asked questions

What is the safest way to share an S3 bucket with another AWS account?

Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer’s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.

Do I need both a bucket policy and an IAM policy?

For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.

Why do I get Access Denied when the bucket policy clearly allows the other account?

Work through it in this order: the consumer’s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account’s policies looks wrong.

Can I share data with an account outside my AWS Organization?

You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.

Should I use S3 Access Points or Lake Formation?

It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.

How do I revoke cross-account access cleanly?

Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.

Does cross-account access change anything about cost?

The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.

The one thing to remember

Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.

Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.

Need a second pair of eyes on your cross-account setup?

Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:

• Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.
• Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.
• Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.
• Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.
• Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.
• Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.

Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.

The post AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies appeared first on John Nessime.