That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.

The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.

What this post covers

Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn’t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.

Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.

Failure family one: there is no join key, only a filename

Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.

Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.

What to use instead

Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.

Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for “this exact bytes-on-disk artefact.” If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.

S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.

Failure family two: extraction confidence gets flattened into a fact

Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.

Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn’t go away. You just stopped being able to see it.

Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always “which extractor produced these,” and without that column the answer is a week of archaeology.

-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);

The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.

Failure family three: two catalogs, two permission models

This is the one that bites teams who did everything else right.

Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.

The result is predictable and unpleasant. An analyst who is correctly denied the salary column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.

There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:

• Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.
• Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.
• Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.
• Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.

Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with .metadata.json appended. That file is where classification, tenant, document ID and effective dates belong.

{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}

Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.

Failure family four: index decisions you cannot take back

S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.

By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.

If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.

# Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'

The dimension has to match your embedding model’s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.

Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.

Failure family five: reprocessing rewrites history

You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter’s board pack change.

This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.

Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and “why did this number move” has an answer you can query instead of guess at.

Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.

A shape that works for combining structured and unstructured data on AWS

Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.

1. Land raw, enable versioning, enable S3 Metadata. Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.
2. Mint the document ID at landing. Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.
3. Classify before you extract. Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.
4. Extract to a staging table, not straight to the curated table. Staging gives you somewhere to run validation before anything touches a table an analyst can query.
5. Merge into Iceberg on document identity, carrying confidence and provenance columns. Never merge on object key.
6. Embed separately, keyed by the same document ID. The vector plane and the table plane are different systems with different refresh rates. Treat them as such.
7. Join at query time in Athena. Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.

Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.

Troubleshooting

Athena says the principal has no privilege on the resource

This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.

Queries against metadata tables fail with a syntax error

Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.

Knowledge base sync fails partway with validation errors

Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.

Retrieval returns documents the caller should not see

Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.

Athena gets slower week over week on a table that isn’t growing much

Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.

Common mistakes

• Joining on the S3 object key because it was there and it worked in the prototype.
• Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.
• Applying a confidence threshold and discarding the score.
• Creating the vector index before deciding on the embedding model.
• Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.
• Reprocessing in place, with no record of which extractor produced which value.
• Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.

Best practices

• One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.
• Confidence and provenance columns are not optional. They are the difference between debugging and guessing.
• Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.
• Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.
• Enforce classification in all three planes independently. There is no single choke point.
• Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.
• Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.

FAQ

Do I need a lakehouse to combine structured and unstructured data on AWS?

No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.

Should extraction output live in the same table as the structured source data?

Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.

Is S3 Vectors a replacement for OpenSearch?

For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.

How do I keep the vector index in sync with the documents?

Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because “mostly in sync” degrades quietly.

What breaks first when this architecture is done badly?

Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.

Can I use Textract instead of Bedrock Data Automation?

Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.

Does Lake Formation govern my documents as well as my tables?

It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.

The one thing worth remembering

Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.

Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.

Need help wiring documents and tables together?

This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:

• Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.
• Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.
• Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.
• Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.
• Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.
• Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.

Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.

The post The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS appeared first on John Nessime.

That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.

This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.

First, understand what you are actually migrating

The framing “SageMaker Lakehouse versus Glue and Athena” is misleading, and it drives bad decisions.

SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.

The unit it introduces is the catalog: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A managed catalog holds data you own in S3 or Redshift Managed Storage. A federated catalog points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.

Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.

So the question is not “should we move our data”. It is “should we move our governance”.

The failure mode that actually bites

Lake Formation grants Super permission to a virtual group called IAMAllowedPrincipals on every existing Data Catalog database and table by default. That group effectively means “anyone whose IAM policy already lets them in”. It exists purely for backward compatibility. While it holds Super on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.

So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke IAMAllowedPrincipals so the grants take effect.

That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.

It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.

Hybrid access mode is the way through

Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.

The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.

The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.

Where the lakehouse genuinely wins

• One permission surface across engines. If governance today is “Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB”, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.
• Tag-based access control across federated sources. LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.
• Querying across boundaries without copying. A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.
• The Iceberg REST interface. Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.
• Cross-account sharing without replication. For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.

Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.

Where Glue plus Athena is still the right answer

• Single account, S3 only, one team. Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.
• Your access control is coarse and that is correct. Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.
• You have heavy IAM-only automation. Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.
• Your bottleneck is query cost, not governance. Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.
• You depend on a specific engine behaviour today. Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.

The cost levers that move, and the ones that don’t

Pricing pages change, so here are the billing mechanisms rather than numbers.

Unchanged: metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.

New line items:

• Lambda invocations behind federated catalogs. When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.
• Service-managed Redshift Serverless compute. Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.
• Redshift Managed Storage. Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.

The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:

# Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -> Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"

Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog’s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.

Is a SageMaker Lakehouse migration worth it? A decision procedure

Run these in order. Stop at the first one that gives a clear answer.

1. Count your non-S3 governed sources. Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.
2. Inventory every principal touching the Data Catalog. Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.
3. Ask whether column or row-level control has actually been requested. If the answer is a compliance requirement with a name attached, proceed. If it is “it would be nice”, you are buying a permission model to solve a hypothetical.
4. Check engine coverage against your real workloads. Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.
5. Price the federated path against a copy. For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.
6. Decide the destination, not just the direction. “Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets” is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.

Three things to run before you commit

Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:

aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }

S3 Tables catalogs use a s3tablescatalog/bucket-name naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on list-data-catalogs will quietly under-report, so enumerate S3 Tables catalogs separately.

Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:

aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'

This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.

Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:

--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true

Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.

Arguments that don’t survive contact

• “It’s the same Glue Data Catalog, so migrating is low risk.” True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.
• “We’ll migrate to avoid falling behind.” AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.
• “Federated catalogs mean we can retire our ETL.” Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.
• “Iceberg REST support means no lock-in.” The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.
• “We can test this in a sandbox.” You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.

How I’d decide

S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.

Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a “clean” fully-enforced state is aesthetic, not operational.

Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.

Frequently asked questions

Does SageMaker Lakehouse replace AWS Glue and Athena?

No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.

Will a SageMaker Lakehouse migration break my existing Athena queries?

Reorganising into catalogs does not break queries on its own. What breaks them is revoking Super from IAMAllowedPrincipals before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.

Do I have to move data into Redshift Managed Storage?

No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.

How much extra does it cost?

The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.

Is SageMaker Unified Studio required?

No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.

What is the smallest useful first step?

Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.

Conclusion

A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.

That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.

Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke IAMAllowedPrincipals until you can name every principal that will notice.

Need help deciding, or unpicking a migration already underway?

An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:

• Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked
• Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage
• Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for
• Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants
• Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later
• Honest second opinions on whether a migration is worth doing at all, including the answer that it isn’t

Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I’ll tell you what I’d do with it.

The post SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It? appeared first on John Nessime.

The dashboard usually isn’t lying. It’s answering a slightly different question, and the cause is almost always the same one: the pipeline recorded what the property management system said on the day it read it, and the PMS has since changed its mind. A reservation got shortened. A rate was corrected. A rebate landed against a folio that closed two weeks ago. The lake never went back to look.

This post is about building a hotel data lake on AWS that survives that. Not a service tour, because the AWS documentation already covers S3, Glue and Athena better than a blog post can. What it covers instead is the handful of modelling and ingestion decisions that determine whether your numbers reconcile with the night audit six months from now, and what each of those decisions costs you.

The failure that shows up three weeks late

Most first attempts at hotel analytics are append-only. Pull yesterday’s reservations, write Parquet under a folder keyed by the day you pulled it, register the partition, done. Yesterday reconciles perfectly. So does the day after. Confidence goes up.

The problem is that hotel source data is mutable in the past, and it stays mutable for a long time. Group blocks get released. Stays get extended at the desk. Adjustments and rebates get posted against nights that already closed. Commission corrections come back from an OTA weeks after checkout. Night audit itself is a rewrite operation: it closes the business date and posts room and tax charges for every occupied room, and if the auditor reruns it or corrects a posting, the figures for that date change after you already stored them.

An append-only lake captures a diagonal. For each business date you keep the first thing you ever saw about it, and nothing after. Every individual day looks right on the day you check it. The month-to-date total drifts. Nobody notices until close, and by then you have no idea which nights moved or why.

That is the whole reason to be careful about grain and table format. Everything else in a hospitality pipeline is ordinary data engineering.

Fix the grain before you fix the pipeline

The core hotel metrics are per-night metrics. Occupancy is rooms sold divided by rooms available. ADR is room revenue divided by rooms sold. RevPAR is room revenue divided by rooms available, which is the same as ADR multiplied by occupancy. All three are defined against a single night.

What the PMS hands you is reservations. A reservation spans nights, carries one confirmation number, and may have different rates on different nights of the same stay. If you store reservations and try to compute occupancy from them, you end up writing date-range arithmetic into every query, and two analysts will write it differently.

Explode to room-nights once, in the pipeline

The fact table that makes everything else easy has one row per room, per night, per reservation. Stay dates, rate code, market segment, source of business, channel, room type, rate amount and the taxes as separate columns. Every downstream metric becomes a filter and a sum instead of a range join.

Do the explosion in the transform layer, not in the BI tool. If a range join lives in a Power BI model or a QuickSight dataset, it exists once per report and drifts once per report.

The denominator is a decision, not a fact

“Rooms available” sounds like a constant. It isn’t. Rooms out of order for renovation, rooms out of service for a night, house-use rooms and comp rooms all get treated differently by different properties, and the PMS report you’re reconciling against has already made a choice.

Same question on the numerator. Whether no-show revenue counts as room revenue, whether cancellation fees count, whether package components get split out of the room rate, whether ADR is quoted net or gross of tax. Hotel accounting conventions follow the Uniform System of Accounts for the Lodging Industry, and most PMS vendors document how their own reports treat each case. Read that documentation before you write a single SQL definition.

Write the definitions down in the repository next to the transform code. When the lake and the PMS disagree later, the first question is always whether it’s a bug or a definition, and without written definitions you cannot answer that question quickly.

Model restatements with two dates, not one

Every row needs a business date, which is the night the revenue belongs to, and an as-of timestamp, which is when you learned it. Two time axes. That’s what makes the difference between “the numbers changed” and “here is exactly what changed, when, and by how much”.

There are two ways to store that. Keep every version of every row and let the reader pick a version, or keep the current truth in a table you update in place and keep the history separately. The second is easier to query and easier to explain, and it’s the one I reach for first. It needs a table format that supports updates and deletes, which in practice on AWS means Apache Iceberg.

Iceberg gives you three things that matter here: row-level MERGE, snapshots you can query as of a point in time, and schema evolution without rewriting the dataset. Athena, Glue ETL, EMR and Redshift all read Iceberg tables through the Glue Data Catalog, so you’re not tying the storage to one engine.

A curated room-night table in Athena looks roughly like this. Note the partitioning, which I’ll come back to.

CREATE TABLE curated.room_night (
  property_code      string,
  business_date      date,
  confirmation_id    string,
  room_number        string,
  room_type_code     string,
  rate_code          string,
  market_segment     string,
  source_of_business string,
  room_revenue       decimal(12,2),
  room_tax           decimal(12,2),
  reservation_status string,
  guest_key          string,
  as_of_ts           timestamp,
  source_system      string
)
PARTITIONED BY (property_code, month(business_date))
LOCATION 's3://example-lake-curated/room_night/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

Loading is a MERGE from a staging table holding the latest extract. Matched rows get overwritten with the newer version, unmatched rows get inserted, and rows that vanished from the source get marked rather than deleted.

MERGE INTO curated.room_night AS t
USING staging.room_night_batch AS s
  ON  t.property_code   = s.property_code
  AND t.business_date   = s.business_date
  AND t.confirmation_id = s.confirmation_id
  AND t.room_number     = s.room_number
WHEN MATCHED AND s.as_of_ts > t.as_of_ts THEN
  UPDATE SET
    room_revenue       = s.room_revenue,
    room_tax           = s.room_tax,
    rate_code          = s.rate_code,
    reservation_status = s.reservation_status,
    as_of_ts           = s.as_of_ts
WHEN NOT MATCHED THEN
  INSERT (property_code, business_date, confirmation_id, room_number,
          room_revenue, room_tax, rate_code, reservation_status, as_of_ts)
  VALUES (s.property_code, s.business_date, s.confirmation_id, s.room_number,
          s.room_revenue, s.room_tax, s.rate_code, s.reservation_status, s.as_of_ts);

The as_of_ts comparison in the MATCHED clause is doing real work. Extracts arrive out of order more often than you’d like, particularly when a backfill runs alongside the scheduled job, and without that guard a late-arriving older snapshot overwrites newer truth. It’s a silent corruption, which is the worst kind.

Keep a restatement log, not just the current state

Before the MERGE runs, write the delta out to an append-only table: key, field changed, old value, new value, as-of timestamp. It costs almost nothing and it turns “the March numbers moved” into a query that returns rows.

You can also lean on Iceberg snapshots and time travel for this, and they’re useful for spot checks. I still prefer an explicit log, because snapshot retention will eventually expire the snapshot you want, and finance will eventually ask about a night that’s older than your retention window.

Landing the sources without building a point-to-point mess

A property runs more systems than people expect. PMS, POS, channel manager, revenue management, spa and golf, parking, loyalty, and whatever the marketing team bought last year. Land each of them raw and immutable first, then transform. The raw zone is your audit trail and your ability to reprocess without going back to a vendor API.

The PMS is the hard one

For OPERA Cloud, the Oracle Hospitality Integration Platform is the supported route. It exposes REST APIs over OAuth, and registration is self-service rather than going through the older partner validation process. Two things to plan for: the API surface is large and reservation, financial and configuration data live in different areas, and access depends on the property’s licensing, so confirm what the property actually has before you scope the work.

Legacy on-premise OPERA is a different world. Integration there goes through the older interface stack and the validation programme, and scheduled report exports to a file drop are frequently the pragmatic answer. Cloud-native systems like Cloudbeds and Mews expose REST APIs directly and are far less work.

Reading the PMS database directly, where you can even reach it, is tempting and usually a bad idea. Schema changes arrive with vendor upgrades and nothing tells you in advance, and depending on the contract it can put support out of scope. Check the agreement before you build anything on top of direct database access.

The relay box, if you need one

Properties with on-premise systems usually need something that picks up scheduled exports and pushes them to S3. Keep it boring: a small always-on Linux host, a scheduled job, checksum on write, and an SNS or CloudWatch alarm when a file doesn’t turn up. A cheap VPS from somewhere like Contabo or InterServer does this fine, and it is much easier to hand over at the end of an engagement than a machine sitting under the front desk.

For reaching that host without opening inbound ports on a hotel network, an outbound-only tunnel such as Cloudflare Tunnel is a clean fit. If staff are hitting admin interfaces from guest or public Wi-Fi, a consumer VPN client like NordVPN or Surfshark protects that transport, though it is not a substitute for a real network path into the property.

Time zones will bite you

The hotel business date is not a calendar day and it is not UTC. It’s a property-local concept that rolls at night audit, which might be at 2am or 4am depending on the property, and it can be held open when the audit runs late.

Store the source system’s own business date as the partition key. Store event timestamps in UTC alongside it. Never derive the business date by truncating a UTC timestamp, because for a group with properties in several time zones that quietly shifts revenue between nights, and the error is small enough to look like noise.

Keep card data out and plan for erasure on day one

Guest records are personal data and hotels hold a lot of it: names, addresses, passport and ID details in some jurisdictions, loyalty identifiers, stay history. Payment card data sits under a separate regime again.

The single highest-value decision is negative. Do not bring cardholder data into the analytics platform at all. Nothing in occupancy, ADR or segment analysis needs a card number, and pulling one in drags the whole lake, its query engines and everyone with access into PCI DSS scope. Take the payment token or a masked reference from the PMS and stop there.

For guest identity, pseudonymise at ingestion. Keep a surrogate guest key in the analytics tables and hold the mapping to real identity in one small, separately governed place with its own access controls. Amazon Macie can scan the raw zone for personal data that shouldn’t be there, which is useful precisely because someone will eventually land a CSV export nobody reviewed.

Erasure needs thinking about before you have to do it. Iceberg makes row-level deletes straightforward, but a deleted row is still present in older snapshots, and time travel will happily return it. Deletion isn’t complete until snapshot expiration has run past the snapshots containing that row and the underlying files have been removed. Set retention deliberately, know what your window is, and be able to state it. Crypto-shredding, where the identity mapping is encrypted with a per-subject key you can destroy, is the cleaner design if the requirement is strict.

The same discipline applies to workstations. Analysts pull extracts, extracts contain guest names, and laptops get replaced. A secure erasure tool such as O&O SafeErase covers that end of it, and it belongs in the same policy as the lake.

For access control, Lake Formation is the right layer. Define column and row filters once and they’re enforced across Athena, EMR and Redshift rather than reimplemented per engine. A group operator can then give each property’s management team row access to its own data from the same tables, which is far less work than building a schema per property.

File physics at hotel scale

Hotel data is small. A single property produces a few hundred room-nights a day. Even a large group is nowhere near the volumes that data lake tutorials assume, and that changes the advice in one specific direction: your problem is too many tiny files, not too much data.

Two habits create it. Partitioning by day when a property generates kilobytes per day, and polling the PMS every fifteen minutes so each run writes another sliver. Both feel careful and both make queries slower and more expensive, because the engine spends its time opening files rather than reading them.

• Partition curated tables by property and month, not by day. Iceberg prunes from its own metadata, so you get file-level skipping without the partition explosion.
• Batch the writes. Buffer frequent polls and MERGE on a schedule that matches how the numbers are actually consumed, which for most operational reporting means hourly at most.
• Run compaction. The Glue Data Catalog offers managed table optimizers for Iceberg tables covering compaction, snapshot retention and orphan file deletion, configurable from the console, CLI or API. Check the current format and engine support against the AWS docs before you rely on it.
• Or drive it from Athena directly with OPTIMIZE to rewrite data files and VACUUM to expire snapshots and remove orphans.

For the raw zone, where tables are usually plain Hive-style Parquet rather than Iceberg, Athena partition projection is worth knowing about. It computes partition values from a configured pattern instead of listing them from the catalog, which removes a metadata round trip on tables with many partitions. It does not apply to Iceberg tables, which handle pruning through their own manifests.

Athena bills on data scanned and Glue on job runtime, so cost tracks file layout and query shape rather than how much data you own. That’s a good thing at this scale, but it does mean a single badly written dashboard query on a schedule can cost more than the entire storage bill. Tools like Vantage or CloudZero make that visible per workload if the AWS cost console isn’t giving you enough granularity.

Troubleshooting a hotel data lake on AWS

The lake and the PMS report disagree by a small, consistent amount. Almost always a definition, not a bug. Compare the room count first: if rooms sold matches and revenue doesn’t, you’re including or excluding tax, packages or fees differently. If rooms sold doesn’t match, check house-use and comp rooms.

Yesterday reconciles, last month doesn’t. The append-only failure. Query your restatement log for that period. If you don’t have one, compare the current curated table against an older Iceberg snapshot for the same business dates.

Numbers went backwards after a backfill. An older extract overwrote a newer one. Check that the as-of guard is present in the MERGE and that the staging table is deduplicated to one row per key before the merge runs, keeping the highest as-of value.

Occupancy exceeds one hundred percent. Usually duplicate room-nights from a reservation that was modified into a new record while the old one stayed active, or a room moved mid-stay and counted in both rooms for the same night. Add a uniqueness check on property, business date and room number to the pipeline and fail the load rather than publishing it.

Queries got slow and nothing changed. Look at file counts before anything else. Compaction that stopped running, or a new high-frequency job appending small files, explains most of these.

A daily load silently produced nothing. This is the one that hurts, because an empty successful run looks like a healthy pipeline. Alert on the absence of expected rows per property per business date, not just on job failure. A deadman check in CloudWatch, or in Grafana Cloud if you already run dashboards there, catches it the same morning instead of at month end.

Common mistakes

• Storing reservations instead of room-nights, then rebuilding the date logic in every report.
• Treating the ingestion date as the business date. They diverge the first time a load fails and gets rerun the next morning.
• Building the reporting layer before agreeing the metric definitions with the finance and revenue teams.
• Pulling the full guest profile because the API returns it, rather than the fields the analysis needs.
• Skipping the raw zone and transforming on ingest, which means a logic bug is unrecoverable without re-extracting from the vendor.
• Partitioning by day out of habit, then wondering why a small dataset queries slowly.
• Relying on time travel as the audit trail, and discovering retention expired the snapshot finance asked about.

Best practices worth the effort

• One row per room, per night, per reservation. Everything else derives from it.
• Business date and as-of timestamp on every fact row, always.
• Raw zone immutable, curated zone merged, consumption zone shaped for the tool that reads it.
• Metric definitions in version control, next to the SQL that implements them.
• A daily reconciliation query against one PMS report, checked automatically, with a tolerance and an alert.
• Pseudonymise guest identity at ingestion and keep the mapping somewhere separate.
• Compaction and snapshot retention configured before the table grows, not after someone complains.

Frequently asked questions

Do I need Iceberg, or is plain Parquet enough?

Plain Parquet is fine for anything genuinely append-only, such as raw extracts and event logs. It stops being fine the moment you need to correct history, because updating a row means rewriting whole partitions and coordinating that yourself. Since correcting history is the defining characteristic of hotel data, the curated layer wants a table format with row-level updates.

Should the data lake match the PMS exactly?

It should match one named PMS report, on one named metric, within a tolerance you’ve agreed. Trying to match every report is a trap, because vendor reports use different denominators and different treatment of taxes and fees, and several of them disagree with each other. Pick the report finance already trusts and reconcile to that one.

How often should the pipeline pull from the PMS?

Match the decision it feeds. Financial reporting is a daily cycle anchored on night audit and needs nothing faster. Operational views like arrivals and room status benefit from intraday refresh, but that’s a different table with a different retention policy, not a reason to run the whole lake hourly.

Can guest names go in the data lake?

Sometimes, with a lawful basis, retention limits and access controls. But most of the analysis people actually want, occupancy, rate, segment, channel mix, pace, works perfectly on a pseudonymous key. Start without names and add them back only for a use case that genuinely requires them. Card data is a different matter and the answer there is simply no.

Athena or Redshift for the reporting layer?

At single-property and small-group volumes, Athena over Iceberg is usually enough, and there’s no cluster to size or keep warm. Redshift earns its place when you have high-concurrency BI users, heavy joins across many domains, or an existing warehouse investment. Either way, keep the storage open so the choice stays reversible.

How do I handle a group with many properties?

One set of tables with property code as the leading partition key, and Lake Formation row filters for per-property access. Separate schemas per property look tidy at three properties and become unmanageable at thirty, because every schema change and every metric fix has to be applied thirty times.

Where does benchmarking data fit?

Land it in the same lake as a separate domain, at whatever grain the provider supplies, usually a market aggregate rather than a room-night. Keep it in its own tables and join at report time. Mixing external benchmark figures into your own fact table is how index numbers end up in a revenue total.

The one thing to remember

A hotel data lake on AWS is not hard because of the volume. It’s hard because the past keeps changing, and an architecture that assumes otherwise fails quietly and slowly rather than loudly and fast.

Get the room-night grain right, carry a business date and an as-of timestamp on every row, merge instead of appending, and write down what each metric means before anyone builds a dashboard on it. The S3, Glue, Iceberg and Athena assembly is the easy part, and it’s well documented. Reconciling with the night audit six months later is the part that decides whether anyone trusts the thing.

Working on this with me

Most of the hospitality data work I get asked about is one of a small number of shapes. Here’s where I’d be useful:

• Designing the room-night model and the bitemporal load, including the MERGE logic and the restatement log, so your numbers reconcile with the night audit and you can show why they moved.
• Getting data out of the PMS reliably, whether that’s OHIP against OPERA Cloud, a REST API on a cloud-native system, or a scheduled export relay for an on-premise property.
• Auditing an existing lake that has quietly drifted, finding which business dates restated and which metric definitions diverged from the reports finance uses.
• Cutting Athena and Glue spend on a small dataset that behaves like a big one, usually through partition layout, compaction and killing scheduled queries nobody reads.
• Setting up Lake Formation row and column controls for a multi-property group so each property sees its own data from shared tables.
• Building the boring safety net: freshness and row-count checks per property per business date, deadman alerts, and a daily automated reconciliation against a named PMS report.

If you’ve got a discrepancy you can’t explain, send me the two figures and the query behind one of them, and I’ll tell you which side is wrong.

The post Building a Hotel Data Lake on AWS That Agrees With the Night Audit appeared first on John Nessime.

It is a one-line change. It works on the first try. And it is the grant most likely to still be sitting there long after that analytics project shipped, covering a wider set of identities than the person who wrote it ever had in mind.

That asymmetry is the whole problem. A denied cross-account request is a nuisance: someone pings you, you read the policy, you fix it before lunch. A cross-account grant that succeeds, keeps succeeding, and quietly reaches further than intended produces no error, no alert, and no ticket. It surfaces months later during an audit, when someone asks who has been reading that prefix and the honest answer is “anyone the other account decided to let in.”

This post covers AWS cross-account data sharing as a set of failure families rather than a service tour. How the delegation model actually behaves, where encryption keys break or widen access without warning, who ends up owning the bytes after a handoff, why a copy is worse than a reference, and which guardrails catch the mistake you are eventually going to make. Real policy snippets and commands throughout, plus the denials that are hardest to read.

AWS cross-account data sharing is a delegation, not a permission

Start with the mechanic that everything else hangs off. For a request that crosses an account boundary, two separate accounts have to say yes. The resource-based policy in the account that owns the data has to allow the caller, and an identity-based policy in the calling account has to allow the action. Neither side alone is enough.

That is why “I added them to the bucket policy and it still fails” is such a routine ticket. The producer did their half. Nobody attached the matching IAM policy on the consumer side.

Now the part that bites. Look at how most cross-account bucket policies get written:

{
  "Sid": "AllowAnalyticsAccount",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ]
}

Despite how it reads, that :root suffix does not mean the root user. It means the account. You have not granted access to a principal at all. You have delegated the guest list to the administrators of account 111122223333, and they can hand that access to any principal they like, including a role somebody creates next quarter for a purpose you will never hear about.

Sometimes that delegation is exactly what you want, because the consumer team manages its own roles and you do not want to be in the loop for every change. Usually you want the narrower version:

{
  "Sid": "AllowAnalyticsRoleOnly",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["s3:GetObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::shared-events-bucket",
    "arn:aws:s3:::shared-events-bucket/exports/*"
  ],
  "Condition": {
    "StringEquals": { "aws:PrincipalOrgID": "o-abc123def4" }
  }
}

Both halves earn their place. The named ARN stops the wrong role in the right account. The organization condition stops the right role name in the wrong account, which matters if the statement is ever copied, templated, or widened by someone in a hurry. One protects against internal drift, the other against the policy escaping your organization entirely.

Be clear-eyed about the trade-off, though. Naming ARNs is precise and brittle: every new consumer role is a policy change, a pull request, and a review. Organization-wide conditions scale beautifully and narrow nothing inside a member account. If your organization contains twelve accounts and forty teams, aws:PrincipalOrgID on its own is a fairly wide door.

The grant that stops working after a rebuild

When you name a role ARN as a principal, IAM stores the role’s internal unique ID behind the scenes. Delete that role and the policy will start displaying an opaque AROA... string where the ARN used to be, because the ARN no longer resolves to anything. Recreate a role with the identical name and the access does not come back, because the new role has a new unique ID.

A terraform destroy followed by terraform apply on the consumer side reproduces this perfectly. If you are staring at a bucket policy that contains a principal starting with AROA, that is your answer. Rewrite the statement with the current ARN.

The encryption boundary nobody tests until it is live

Second failure family: the key. Encrypting shared data is the easy decision. What people miss is that a KMS key is a second, fully independent authorization surface, with its own resource policy, evaluated separately from the bucket.

• Downloading an object encrypted with SSE-KMS requires kms:Decrypt on the key.
• Uploading one requires kms:GenerateDataKey.
• A multipart upload requires both.

The AWS managed key for S3 cannot carry you here, because you cannot edit its key policy. Sharing KMS-encrypted objects across accounts means a customer managed key, with a statement naming the consumer:

{
  "Sid": "AllowAnalyticsRoleToDecrypt",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/analytics-reader" },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
  }
}

The kms:ViaService condition is cheap insurance. It restricts use of the key to requests that arrive through S3 in that region, so a leaked set of consumer credentials cannot call KMS directly and start decrypting ciphertext it obtained some other way. And the two-key rule applies to the key exactly as it does to the bucket: the consumer’s own IAM policy has to allow the KMS actions too.

Now the sharp edge, which is worth reading twice. If a KMS key is specified by alias rather than by fully qualified ARN, the alias is resolved in the requester’s account. In a cross-account write that means data landing in your bucket can end up encrypted under a key belonging to the writer. You own the object and cannot read it, and no error is raised at write time. Use full key ARNs in anything that crosses an account boundary.

Who owns the bytes after the handoff

Third family: ownership, and it depends entirely on which direction the data moves.

New buckets are created with Object Ownership set to Bucket owner enforced. ACLs are off, and the bucket owner owns every object written into the bucket regardless of who wrote it. That default removed the single most confusing trap in S3 sharing.

Buckets created before that default are the ones to check. If a bucket still sits on the Object writer setting, an object uploaded by another account is owned by that account, and you, holding the bucket, may not be able to read it. Nothing fails at write time. It fails later, when a job in your own account tries to read your own bucket and gets a 403.

Check and fix it with the ownership controls API:

aws s3api get-bucket-ownership-controls --bucket shared-events-bucket

aws s3api put-bucket-ownership-controls 
  --bucket shared-events-bucket 
  --ownership-controls '{"Rules":[{"ObjectOwnership":"BucketOwnerEnforced"}]}'

Order matters when you make that change on a live bucket. Once ACLs are disabled, any PUT that still carries an ACL is rejected with AccessControlListNotSupported. Requests that send no ACL, or send bucket-owner-full-control, keep working. So the sequence is:

1. Find every writer that sets an ACL, including old scripts using --acl and any SDK call passing an ACL argument.
2. Update those clients so they stop sending one.
3. Confirm from CloudTrail that ACL-bearing PUTs have stopped.
4. Flip the bucket to Bucket owner enforced.

Given a free choice of design, I would rather the consumer pull from a bucket I own than push into a bucket I do not. Pull keeps ownership, lifecycle rules, encryption configuration, and access logging in one account, under one team’s control. Push scatters them.

Copies you cannot recall

Fourth family, and the one with the longest tail. There is a large difference between granting a reference and shipping a copy. A reference can be revoked on a Tuesday afternoon. A copy is simply gone.

Three reference-shaped patterns are worth knowing before you reach for a bulk export.

S3 Access Points for object-level sharing

Instead of a bucket policy that grows a new statement every time a team asks for something, you write the bucket policy once to delegate to access points, then give each consumer their own access point with its own focused policy. The delegation is expressed with the s3:DataAccessPointAccount condition. Access points can also be owned by the consumer’s account, so they manage their own detail without touching your bucket policy.

The gotcha: when a request goes through an access point, both the access point policy and the underlying bucket policy have to allow it. A 403 here is easy to misdiagnose because the bucket policy looks correct and is correct. The access point is the layer that said no.

Lake Formation for table-level sharing

When the thing being shared is a table rather than a prefix, Lake Formation is the better fit, because it lets you grant on databases, tables, columns, and row filters instead of on object paths. Cross-account grants ride on AWS Resource Access Manager, so the recipient gets an invitation that a data lake administrator has to accept.

aws ram get-resource-share-invitations

aws ram accept-resource-share-invitation 
  --resource-share-invitation-arn arn:aws:ram:us-east-1:111122223333:resource-share-invitation/EXAMPLE

After the invitation is accepted, the shared database and tables are visible but still not queryable from Athena or Redshift Spectrum until someone creates a resource link, which behaves roughly like a symlink in the recipient’s Data Catalog. Missing resource links account for most of the “the share went through but I cannot see the table” messages.

One setting to check on the producer side is the cross-account data sharing version. Newer versions collapse many grants into far fewer RAM resource shares, which matters once you are sharing at any scale, and they allow granting directly to IAM principals in another account rather than only at account level. Read the current version with:

aws lakeformation get-data-lake-settings

When a copy really is the answer

Sometimes the partner genuinely needs the data on their own infrastructure: a warehouse you have no reach into, a VPS at a provider like Contabo or InterServer, or in the worst case an analyst’s laptop. That is a legitimate outcome, but be honest with yourself and with them that your controls stop at the handoff.

Practical hygiene for that case: scope the extract to the columns actually needed rather than the whole table, put an expiry date in the agreement rather than in your head, and write down what happens to the copy when the engagement ends. If the copy lives on machines you or the client control, verified erasure with something like O&O SafeErase reads a great deal better in an audit than “we deleted the folder.” And if the recipient has no AWS account at all, a time-limited presigned URL is usually a better answer than minting an IAM user with long-lived keys.

The guardrail that catches the mistake you will make

Everything above describes what a careful engineer does per grant. Guardrails are what happens when somebody is not careful, which over a long enough window includes all of us.

Resource control policies are the strongest lever if you run AWS Organizations. An RCP sets a ceiling on what any resource-based policy inside the organization can grant. The canonical statement is “no principal outside my organization may access S3 buckets in my accounts, regardless of what an individual bucket policy says.” That inverts the risk model: a careless bucket policy is no longer an exposure, it is just an ineffective statement.

Two limits are worth stating plainly. RCPs cover a specific set of services rather than everything in AWS, so check the current supported list rather than assuming coverage. And they constrain your resources, not a partner’s behavior once data has legitimately left. An RCP is a wall around your side of the fence.

Service control policies do the mirror-image job on the consumer side, limiting which external accounts your own principals are allowed to assume roles into. Account-level S3 Block Public Access remains a sensible floor underneath both.

For sharing with a third party rather than an internal account, the external ID is not optional in my book:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::444455556666:role/vendor-connector" },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringEquals": { "sts:ExternalId": "a-value-the-vendor-generated-for-you" }
  }
}

The external ID exists because of the confused deputy problem. A vendor holds credentials that can assume roles across many customers, and without a per-customer secret in the condition, customer A can trick the vendor into pointing that access at customer B. If a vendor asks you to create a role for them with no external ID and no conditions at all, that is worth a conversation before it is worth a role.

Verifying that the grant is what you think it is

Reading policies is not the same as knowing what they permit. IAM Access Analyzer answers the question “what is currently reachable from outside my zone of trust” using automated reasoning rather than pattern matching, which is why it catches combinations a human reviewer skims past.

aws accessanalyzer create-analyzer 
  --analyzer-name external-access 
  --type ACCOUNT

aws accessanalyzer list-findings 
  --analyzer-arn arn:aws:access-analyzer:us-east-1:999988887777:analyzer/external-access

Scope the zone of trust to your organization where you have one, then archive the findings that represent intentional shares, so that a new entry in the list actually means something.

The same service exposes custom policy checks you can run in a pipeline before a policy change merges, including checks for public access, for specific actions being granted, and for whether an updated policy grants more than the one it replaces. One thing to understand about them: they evaluate the document you hand them and nothing else. They do not read your account state, so a public-access check can fail on a policy in an account that has Block Public Access enabled. For a merge gate that is correct behavior, because the policy should not be relying on a distant safety net.

Finally, CloudTrail tells you who actually used a grant, which is the input to the only cleanup that ever really works: delete the statements nobody has exercised. Routing that history into CloudWatch, or into a Grafana Cloud dashboard alongside the rest of your platform telemetry, turns “is this still needed” from an argument into a query.

Troubleshooting the denials that are hard to read

• You can list the bucket but every GET returns 403. That pattern points at KMS, not S3. Listing does not touch the key; reading does. Check kms:Decrypt on both the key policy and the consumer’s identity policy.
• 403 through an access point while the bucket policy is clearly correct. The access point has its own policy and it also has to allow the calling principal. Read that one next.
• AssumeRole is denied although the trust policy names the caller. Cross-account needs both halves: the caller’s identity policy must permit sts:AssumeRole against that role ARN. Then check the external ID value and any SCP or permissions boundary in the calling account.
• A principal in the policy displays as AROA followed by random characters. The referenced role was deleted. Recreating it under the same name will not restore access.
• A Lake Formation share appears in the catalog but queries fail. Usually a missing resource link in the recipient account, or a RAM invitation that was never accepted.
• Access worked yesterday and stopped today with nothing changed locally. Look at the organization layer before re-reading the bucket policy. A newly attached RCP or SCP denies in a way that no amount of resource-policy editing will fix.

Common mistakes

• Granting to an account principal when a role ARN would have done, then never revisiting it.
• Treating aws:PrincipalOrgID as least privilege. It narrows the organization, not the account.
• Sharing a bucket and forgetting the key, then debugging S3 for an hour over a KMS denial.
• Referring to KMS keys by alias in cross-account configuration instead of by full ARN.
• Letting a partner push into your bucket without confirming the ownership setting first.
• Exporting a full table when the consumer needed four columns, because the export was easier to write.
• Creating a vendor role with no external ID and no conditions because the vendor’s setup guide said to.
• Granting access with no expiry, no owner, and no note in the policy Sid explaining why it exists.

Best practices worth the effort

• Give every cross-account grant a descriptive Sid. It is the only free documentation slot IAM offers, and future-you will read it.
• Prefer one dedicated consumer role per relationship over a shared role used by several integrations. Revocation becomes a single delete.
• Pair a specific principal with a broad condition, rather than relying on either alone.
• Put policy checks in CI so a widened grant fails a pull request instead of a quarterly review.
• Keep an inventory of active shares with an owner and a review date, even if it is a table in a repository README.
• Enable an external access analyzer scoped to your organization and treat new findings as work, not noise.
• Watch cross-region and internet data transfer when a consumer sits somewhere unexpected. Cost visibility tooling such as Vantage or CloudZero will attribute it faster than reading a bill.

Frequently asked questions

What is the safest way to share an S3 bucket with another AWS account?

Name a specific role ARN in the bucket policy, add an organization or account condition alongside it, scope the resource to a prefix rather than the whole bucket, and make sure the consumer’s identity policy is equally narrow. If the bucket is encrypted with a customer managed key, grant on the key as well. If the list of consumers is going to grow, move to access points rather than growing the bucket policy.

Do I need both a bucket policy and an IAM policy?

For cross-account access, yes. The resource-based policy in the owning account and the identity-based policy in the calling account both have to allow the action. That is the single most common reason a correct-looking bucket policy still returns Access Denied.

Why do I get Access Denied when the bucket policy clearly allows the other account?

Work through it in this order: the consumer’s identity policy, then the KMS key policy if the objects are encrypted with a customer managed key, then any access point policy in the path, then organization-level SCPs and RCPs. Denials from the organization layer are the ones people waste the most time on, because nothing in either account’s policies looks wrong.

Can I share data with an account outside my AWS Organization?

You can, and organization conditions will not help you there. Use a named role ARN, an external ID on any assumed role, a tightly scoped resource, and a defined end date. Also confirm whether an RCP in your organization already blocks external principals for that service, because that is a common reason a partner grant refuses to work no matter how carefully it is written.

Should I use S3 Access Points or Lake Formation?

It depends on the unit of sharing. If the consumer wants objects under a prefix, access points are simpler and cheaper to operate. If the consumer wants a table, and especially if you need column or row filtering, Lake Formation is the right tool and access points would force you to fake fine-grained control with prefix layout. Plenty of environments run both.

How do I revoke cross-account access cleanly?

Remove the statement from the resource policy, remove the KMS key grant, and check for any access point, RAM share, or resource link that also carries permission. Then confirm with CloudTrail that the principal has stopped appearing. Revocation feels done long before it is done, which is exactly why an inventory of active shares pays for itself.

Does cross-account access change anything about cost?

The request charges land on the bucket owner by default, and data transfer depends on whether the consumer is in the same region. A consumer that quietly starts scanning a large prefix from another region is a billing event as much as a security one, which is a good argument for reviewing access patterns and spend together rather than in separate meetings.

The one thing to remember

Getting AWS cross-account data sharing right is less about knowing which service to reach for and more about noticing which grants are quietly larger than they read. An account principal delegates your guest list. An organization condition narrows the wrong dimension. A copy leaves your perimeter permanently and no policy edit brings it back.

Grant references rather than copies, name principals rather than accounts, remember that the key is a second door, and put something in place that watches the perimeter for you. The cross-account grant worth worrying about is never the one that failed. It is the one that worked, and kept working, long after anybody was still looking at it.

Need a second pair of eyes on your cross-account setup?

Most of the cross-account problems I see are not exotic. They are a policy written in a hurry two years ago that nobody has read since. If that sounds familiar, here is where I can help:

• Auditing existing bucket, key, and role trust policies to find grants that are wider than intended, and rewriting them to named principals with conditions.
• Designing a producer and consumer pattern for a specific data handoff, including whether access points, Lake Formation, or a plain scoped role is the right shape.
• Setting up IAM Access Analyzer with a sensible zone of trust and archive rules, so the findings list stays worth reading.
• Adding policy checks to your pipeline so a widened grant blocks a merge rather than surviving to production.
• Building an organization-level perimeter with resource control policies and service control policies without breaking the integrations you already depend on.
• Untangling stubborn cross-account denials, including the KMS, access point, and organization-layer cases that hide well.

Send me a bucket policy, a trust policy, or the CloudTrail entry for the denial you cannot explain, and I will tell you what it actually permits.

The post AWS Cross-Account Data Sharing Done Safely: Share Access, Not Copies appeared first on John Nessime.

It is quick. You can have a working connection inside an hour. What isn’t quick is everything that surfaces three weeks later, when forty people are refreshing dashboards on a schedule nobody on your team controls and the Athena line on the invoice has quietly tripled.

Getting a BI tool to authenticate against AWS and return rows is a solved problem with decent vendor documentation. What is not well covered is what happens afterwards. This post covers how to connect BI tools to AWS data in a way that survives contact with real users: the four access patterns worth considering, why fine-grained security and cheap dashboards pull in opposite directions, how identity actually flows through a shared service account, and the cost levers in the order they pay off.

The failure that shows up on the invoice, not in the logs

Start with the shape of the problem, because it explains most of the design decisions that follow.

A dashboard is not one query. Every tile is a query. A twelve-tile dashboard opened by forty analysts, with an hourly scheduled refresh on top, is a query generator running against your data lake. Athena bills on bytes scanned, so the cost of that pattern is set almost entirely by how much data each tile touches, not by how “small” the answer looks in the browser.

Athena has a feature built for exactly this. Query result reuse lets a repeated query return the previous stored result instead of scanning again, within a maximum age you choose. If you don’t specify an age, Athena treats results older than 60 minutes as expired, and the longest age you can specify is the equivalent of seven days.

Here is the part that catches people, and it is the whole reason I wrote this post.

Result reuse is not available for tables that have Lake Formation row or column filters. It is also unavailable when the table’s S3 location is registered as a Lake Formation data location, for queries that reference more than one data catalog, for queries referencing more than twenty tables, and for federated catalogs or an external Hive metastore.

So the sequence goes like this. Dashboards are connected and cheap. A security review asks for row-level security on the customer dimension. You add Lake Formation data filters, which is exactly the right call. Everyone signs off. And the reuse rate on those tables silently drops to zero. No error. No alert. No code change. Just a larger bill the following month and a very awkward conversation about what changed.

You can still have both governance and cheap dashboards. You just can’t have both on the same table. The usual escape is to materialise pre-filtered, per-audience tables that carry no filters of their own, and grant access to those instead. That costs you a pipeline and some storage. It buys back the caching.

Pick the door before you pick the driver

Most guides start with driver installation. That’s backwards. The driver is a twenty-minute job once you’ve decided which door the BI tool walks through. There are four realistic doors, and they fail in different ways.

Athena over the AWS Glue Data Catalog

The default, and usually the right first answer. Files stay where they are, the Glue Data Catalog is the single source of truth for schema, and anything that speaks JDBC or ODBC can connect. Tableau, Power BI, Looker, Metabase, Apache Superset and Grafana all handle this without exotic configuration.

Where it wins: no cluster to size, no idle spend, and you can expose a new dataset by cataloguing it rather than by loading it anywhere.

Where it doesn’t: per-query billing punishes chatty dashboards, and interactive latency is measured in seconds rather than milliseconds. If your users expect a filter dropdown to respond instantly, they will be disappointed and they will tell you so.

One practical note that saves an afternoon: for Tableau Desktop, the Athena JDBC driver has to be dropped in a specific directory before the connector appears. On macOS that’s ~/Library/Tableau/Drivers, and on Windows it’s C:Program FilesTableauDrivers. If the analyst says “I don’t see Amazon Athena in the list,” that’s almost always why.

A Redshift datashare into a consumer warehouse

If the data already lives in Redshift, or if concurrency is the binding constraint, a datashare is the cleaner answer. The producer keeps ownership; the consumer gets live, transactionally consistent access and pays with its own compute, so a badly written dashboard can’t degrade your ETL warehouse.

-- On the producer
CREATE DATASHARE bi_share;
ALTER DATASHARE bi_share ADD SCHEMA reporting;
ALTER DATASHARE bi_share ADD TABLE reporting.fact_orders;
GRANT USAGE ON DATASHARE bi_share TO ACCOUNT '111122223333';

The schema has to be added before the tables inside it, which is a common first stumble. Cross-account sharing is a two-way handshake: the producer authorises the datashare, and an administrator in the consumer account associates it before anything is queryable. Both sides must be encrypted for cross-account sharing. Objects are read-only unless the producer explicitly grants write privileges and the consumer associates for writes, which is the behaviour you want for a BI audience.

Two things to price in. Cross-region sharing moves data and is billed accordingly. And the consumer warehouse doesn’t hold statistics on the producer’s data, so plan expectations around query planning rather than assuming parity with local tables.

The Iceberg REST catalog

If the consumer is another query engine rather than a dashboard tool, this is the interesting door. Amazon S3 Tables exposes an Iceberg REST endpoint at https://s3tables.<region>.amazonaws.com/iceberg, with your table bucket ARN as the warehouse location and SigV4 for authentication. Engines like Snowflake, Trino, StarRocks, Spark and PyIceberg can register that as an external catalog and read the tables in place.

Worth knowing: AWS positions the S3 Tables endpoint for basic read and write access against a single table bucket, and recommends the AWS Glue Iceberg REST endpoint when you need unified table management, centralised governance and fine-grained access control. If Lake Formation is already your control plane, go through Glue.

Where it doesn’t help: most BI tools don’t speak Iceberg REST. They speak SQL over JDBC. This is an engine-to-engine door, not a dashboard door, and treating it as the latter wastes a week.

A materialised extract

The unglamorous option that is frequently correct. Build a narrow, pre-aggregated table on a schedule and land it where the BI tool already keeps data: QuickSight SPICE, a Power BI import model, a Tableau extract, or a small Postgres instance. Tools like dbt handle the modelling side; Fivetran or Airbyte handle it if the source is a SaaS system rather than your own lake.

Where it wins: it is by a wide margin the cheapest and fastest option for stable reporting, because the expensive scan happens once per day instead of once per tile per user. If you self-host Metabase or Superset, a modest VPS from a provider like InterServer or Hetzner will comfortably serve a department off a nightly rollup.

Where it doesn’t: freshness. And you now own a pipeline that can fail silently, which is its own category of on-call pain.

Identity: the shared service role problem

Nearly every BI deployment starts with one IAM role for the whole tool. It works immediately, which is why it survives longer than it should.

The consequences accumulate quietly. CloudTrail records every query as the same principal, so audit answers stop being useful. Lake Formation row filters key off the principal, so if all traffic arrives as one role, your row-level security collapses to whatever that role can see. And cost attribution disappears, so when the bill jumps you cannot say which team caused it.

Three ways out, in increasing order of effort:

1. One role and one Athena workgroup per consumer group. Coarse, but honest. Finance gets a role, marketing gets a role, each with its own workgroup. Scan volume becomes attributable and you can set different limits per audience.
2. Real identity propagation. Federate the BI tool’s users through IAM Identity Center or SAML so Lake Formation grants apply to the actual person. This is the correct boundary and the most work, and driver and tool support varies. Test it with the exact tool version your users have before you promise it in a design review.
3. Push the filter into the data. Per-tenant tables, views or partitions with separate grants. Unfashionable and more to maintain, but it works with every tool, survives driver upgrades, and keeps result reuse eligible because the tables themselves carry no filters.

Option three is the one I reach for first when the audience is a handful of stable groups, and option two when the audience is the whole company.

The network path is a design decision, not a checkbox

For a SaaS BI tool, the public service endpoint with SigV4-signed requests over TLS is perfectly reasonable, and it is what most vendors document. Don’t let anyone talk you into complexity you don’t need.

You need more when the tool runs inside a VPC, or on-premises via Direct Connect or VPN, and traffic must not traverse the internet. Athena supports an interface VPC endpoint, and the Athena documentation is explicit that you should pair it with an AWS Glue endpoint, because the catalog lookup is a separate API call that will otherwise still go out to the public endpoint.

aws ec2 create-vpc-endpoint 
  --vpc-id <your-vpc-id> 
  --service-name com.amazonaws.<region>.athena 
  --vpc-endpoint-type Interface 
  --subnet-ids <subnet-id> 
  --security-group-ids <security-group-id> 
  --private-dns-enabled

With private DNS enabled, the standard athena.<region>.amazonaws.com hostname resolves to the endpoint, so no client configuration changes. Without it, VPC gives you a hostname in the form <endpoint-id>.athena.<region>.vpce.amazonaws.com, which you have to configure in the tool by hand. Attach an endpoint policy while you’re there; that’s where you restrict which principals and resources can be reached.

For Redshift, the equivalent is a Redshift-managed VPC endpoint, which requires either RA3 node types or Redshift Serverless. On provisioned clusters, cluster relocation has to be enabled first.

One cost item people forget: if the BI tool lives outside AWS, every row it pulls back is data transfer out. A live dashboard returning aggregates is cheap. A nightly extract refresh dragging millions of rows across the boundary is not. Add an S3 gateway endpoint for the query results bucket so the driver’s result download stays on the AWS network where it can.

Cost levers, in the order they pay off

Ordering matters here. Most teams reach for the guardrails first, which bounds the worst case without improving the base case.

1. Fix the storage layout. Columnar formats and sensible partitioning reduce bytes scanned on every query forever. Nothing else you do competes with this.
2. Pre-aggregate for the dashboard. A tile that reads a purpose-built rollup is a different animal from one that scans a fact table and aggregates on the fly.
3. Enable result reuse where governance allows it. Free wins on repeated queries, provided the tables aren’t disqualified by the limitations above.
4. Give every consumer its own workgroup with a per-query cutoff. This is the guardrail, and it belongs after the first three, not before them.
5. Move the highest-traffic dashboards to extract mode. Once a dashboard’s usage is predictable, importing beats querying live.

The workgroup guardrail in Terraform. The per-query cutoff cancels any single query that scans more than the limit, which is what stops an accidental SELECT * against a multi-terabyte table:

resource "aws_athena_workgroup" "bi_finance" {
  name  = "bi-finance"
  state = "ENABLED"

  configuration {
    enforce_workgroup_configuration    = true
    publish_cloudwatch_metrics_enabled = true
    bytes_scanned_cutoff_per_query     = 10737418240  # 10 GiB

    result_configuration {
      output_location = "s3://my-athena-results/bi-finance/"
    }
  }
}

Two flags there do real work. enforce_workgroup_configuration stops clients overriding the results location and encryption settings, which is what keeps a BI tool from writing results somewhere unmanaged. publish_cloudwatch_metrics_enabled is what makes the whole thing observable, and without it you’re back to reading the invoice to find out what happened.

Workgroup-wide limits are configured separately from the workgroup resource. Those let you set hourly or daily thresholds on aggregate data scanned and fire an SNS notification when a threshold is crossed, which can page someone or disable the workgroup outright.

Result reuse can be set per query through the API, which is the mechanism the drivers expose as a connection option:

aws athena start-query-execution 
  --work-group "bi-finance" 
  --query-string "SELECT region, SUM(net) FROM reporting.orders GROUP BY 1" 
  --result-reuse-configuration 
    "ResultReuseByAgeConfiguration={Enabled=true,MaxAgeInMinutes=60}"

Set the max age to match how often the underlying data actually changes. If the table is rebuilt once a night, an hour is far too conservative.

Finally, alarm on the scan metrics. Whether that lands in CloudWatch dashboards, Grafana or Datadog matters less than the alarm existing at all, owned by the team whose workgroup it is.

Troubleshooting the connections that half work

• The tool connects but lists no tables. Almost always Lake Formation rather than IAM. Check that the role has been granted on the database and tables, and note that the DESCRIBE permission on the default database is required for Athena.
• “Bytes scanned limit exceeded.” That’s your per-query cutoff doing its job. Look at what the tile is actually querying before raising the limit. Usually the partition filter is missing.
• Query works in the console, fails from the tool. Three usual suspects: a different workgroup, a results location the role can’t write to, or a role that has catalog access but not S3 access to the underlying data.
• A view works for you but not for the analyst. Lake Formation requires permissions on the tables, columns and S3 locations the view is built on. Column-level permissions are not available on views, so a view is not a substitute for column filtering.
• Costs jumped with no deployment. Check whether anyone added Lake Formation filters, and check whether partition pruning quietly stopped working after a schema or layout change.
• The first query of the morning is slow, the rest are fine. Partition metadata listing. Partition projection removes most of that cost on large tables.

Common mistakes

• Granting the BI role broad S3 read access and calling it least privilege. It sees every bucket prefix it can list, catalog or not.
• Running every consumer through the default primary workgroup, which makes attribution and per-team limits impossible.
• Exposing raw fact tables to business users and expecting them to write efficient SQL. They will drag columns into a canvas, which is what the tool is for.
• Setting dashboard refresh schedules without asking how fresh the underlying data actually is. Refreshing hourly against a nightly-loaded table burns money for identical results.
• Promising row-level security through a shared service role. It will not behave the way the security review thinks it does.
• Skipping the Glue VPC endpoint after building the Athena one, then spending a day wondering why traffic still leaves the VPC.

Best practices worth the effort

• Create the workgroup, the role and the tag before you hand over the connection string, not after the first invoice.
• Publish a curated reporting layer. The BI tool should see purpose-built tables with business column names, not your raw landing zone.
• Decide consciously whether a dataset is governed by Lake Formation filters or optimised for caching, and document which, because the two do not coexist on one table.
• Apply cost allocation tags at creation time. Retrofitting them across a live estate is thankless work.
• Version the connection setup as code. Terraform for the workgroup, roles and endpoints; a short runbook for the driver install on the client side.
• Re-test the whole path after any driver upgrade. Authentication behaviour is the thing that changes between driver major versions.

Frequently asked questions

What is the fastest way to connect Power BI or Tableau to AWS data?

Athena over the Glue Data Catalog, using the Athena JDBC or ODBC driver. Catalogue the data, create a workgroup with a results bucket, create a role with catalog and S3 read access, install the driver on the client, connect. The connection itself is the easy part; the workgroup and role are what you’ll wish you had done deliberately.

Should I use Athena or Redshift for BI dashboards?

Athena if query volume is moderate, the data already lives in S3, and users can tolerate a few seconds per query. Redshift if you need high concurrency, consistently low latency, or you already run a warehouse and can share into a separate consumer endpoint. The honest test is whether your peak is a spike or a plateau: Athena handles spikes without idle cost, Redshift handles plateaus without per-query cost.

Does Lake Formation row-level security work with any BI tool?

The filtering itself is enforced by the query engine, so it works regardless of the tool. What varies is whether the tool can present each end user as a distinct principal. Through a single shared service role, every user gets the same filtered view, which is not what row-level security is for. Verify the identity path with your actual tool before designing around it.

Why did my Athena bill increase after enabling fine-grained access control?

Because query result reuse is not supported for tables with Lake Formation row or column filters, or where the S3 location is registered as a Lake Formation data location. Queries that were being served from cached results started scanning again. Nothing errors, which is why it takes a billing cycle to notice.

Do I need PrivateLink to connect a BI tool to AWS?

No, unless your requirements say traffic must not traverse the public internet, or the tool runs inside a VPC or on-premises network. Public endpoints with signed requests over TLS are the normal path for SaaS BI tools. Add interface endpoints when there’s a stated requirement, and remember to add the Glue endpoint alongside the Athena one.

Can BI tools read Amazon S3 Tables directly?

Indirectly. S3 Tables integrates with the Glue Data Catalog, so Athena, EMR and Redshift can query them, and any BI tool that connects through Athena inherits that. The Iceberg REST endpoint is aimed at query engines and Iceberg clients rather than dashboard tools, which generally expect a SQL connection instead.

Is QuickSight a better fit than an external BI tool?

It removes the network, driver and identity plumbing described above, and SPICE gives you an in-memory cache without building an extract pipeline. Against that, your organisation may already be committed to Tableau, Power BI or Looker for reasons that have nothing to do with infrastructure, and retraining an analytics team is a genuine cost. Evaluate it on that basis rather than on the connection mechanics alone.

The one thing to take away

When you connect BI tools to AWS data, the connection is not the project. The project is deciding, deliberately, which datasets are governed by fine-grained filters and which are optimised for repeated cheap reads, then giving every consumer its own role, workgroup and cost tag before anyone gets a connection string.

Do that and a surprise bill becomes a question with an answer. Skip it and you’ll be reading CloudTrail on a Friday afternoon trying to work out which dashboard scanned four terabytes, with no way to prove it.

Need help wiring this up properly?

This is a large part of what I do. If you’re opening up AWS data to an analytics team and want it done so it still makes sense in six months, I can help with:

• Choosing between Athena, a Redshift datashare, an Iceberg REST catalog or a scheduled extract for your specific query pattern and user count
• Setting up per-consumer IAM roles, Athena workgroups, scan limits and cost allocation tags as Terraform rather than console clicks
• Designing a Lake Formation permission model that gives you row and column security without silently destroying your caching
• Building the curated reporting layer and partition strategy that makes dashboards fast and cheap at the source
• Private network paths with interface VPC endpoints, Redshift-managed endpoints and endpoint policies for tools that can’t use public endpoints
• Diagnosing an Athena or Redshift bill that grew without an obvious cause, and putting alarms in place so it doesn’t happen twice

Send me a workgroup config, a slow dashboard query, or a Cost Explorer screenshot you can’t explain, and I’ll tell you what I’d look at first.

The post How to Connect BI Tools to AWS Data Without a Runaway Bill appeared first on John Nessime.