That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.

This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.

SageMaker Unified Studio is a front door, not a runtime

Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.

AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project’s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new authorisation layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.

What it actually replaces

The pile of standalone studios and query editors

This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.

The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a “workspace” meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.

Per-team domain sprawl

The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.

The access request spreadsheet

There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.

The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.

What it doesn’t replace

Your bill, or your ability to explain it

There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.

What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer’s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.

Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.

Your IAM and Lake Formation design

Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.

Nobody in the adoption meeting will say “this fixes our permissions mess.” Plenty of people will quietly assume it. Say it out loud early.

Your infrastructure as code

This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.

It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.

SageMaker AI

The original SageMaker was renamed SageMaker AI and remains available on its own. If your team’s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.

What SageMaker Unified Studio quietly adds

Here is the section that rarely makes it into the migration plan.

You now operate DataZone, intentionally or not

SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a dzd prefix, and the console for creating one still lives under the DataZone console path.

You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:

# Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID

Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.

Project creation is a CloudFormation deploy wearing a friendly hat

Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.

This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation “just doesn’t work,” do not reproduce it in the UI. Read the stack events:

# Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table

The first failed resource causes the rollback. Everything after it is noise. This one habit turns most “the platform is broken” tickets into five-minute answers.

A domain-type fork you will live with for a long time

Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.

Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post’s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.

Where adoption actually goes wrong

“It works in my console but not in the project”

This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.

Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:

aws sts get-caller-identity

If the ARN is not the one you assumed, stop. You have found the problem.

The bill went up after a migration that was supposed to save money

Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.

Nobody can find the data that is definitely there

Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.

Common mistakes

• Treating it as a migration rather than an addition. Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.
• Letting users self-serve projects on day one. Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.
• Choosing the domain authentication type in a hurry. It is the hardest thing here to change later.
• Skipping the catalog. Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.
• Assuming the CI/CD CLI provisions your platform. It deploys applications into projects that already exist.
• Not tagging project resources from the start. Retrofitting attribution across dozens of blueprint-created resources is miserable work.

How I’d decide

A rough order of operations for anyone weighing this up:

1. Ask what problem you are solving. If it is “our teams cannot discover or safely share each other’s data,” this is the right tool. If it is “we have too many consoles,” it is an expensive way to buy fewer bookmarks.
2. Count your teams. The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.
3. Decide the authentication model first. Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.
4. Write the project profiles yourself. The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.
5. Prove the teardown path before the build path. Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.
6. Keep the old consoles. They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.

One more piece of judgement: wire your existing observability in from the start rather than relying on the studio’s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.

Frequently asked questions

Does SageMaker Unified Studio replace SageMaker Studio?

It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.

Do I still need Amazon DataZone separately?

Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.

How much does SageMaker Unified Studio cost?

The interface has no standalone charge. You pay for each AWS service you use through it on that service’s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.

Can I manage domains and projects with Terraform?

AWS’s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.

What happens to my Glue jobs and Athena queries?

Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.

Is it worth adopting for a small team?

Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.

Can I run both the old and new experiences at once?

Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.

The one thing worth remembering

SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.

That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was “fewer tabs.” The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.

Ask which one you are before the first project gets created, not after the fortieth.

Need a second opinion on your adoption plan?

This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:

• Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup
• Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults
• Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use
• Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover
• Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants
• Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long

Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.

The post SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t appeared first on John Nessime.

The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.

I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.

This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.

The bill arrives before the carousel does

Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.

Solution training. New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.

Inference capacity. A campaign has minProvisionedTPS, a domain recommender has minRecommendationRequestsPerSecond. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.

Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.

import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])

Decision one: domain recommender or custom solution

A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.

The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.

Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.

Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.

On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.

One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at aws-hrnn, its advice about cold items and training cadence is stale too.

Decision two: how interaction data gets in

Bulk imports replace unless you tell them not to

This is the one that costs people a month of confused model quality.

A dataset import job defaults to FULL, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is INCREMENTAL, which appends and only replaces records sharing an ID.

Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.

import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)

Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.

Streaming events through an event tracker

The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the personalize-events client.

import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)

Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.

That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.

The sessionId field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.

Decision three: schema and event types, decided once

Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.

{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}

The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.

Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.

Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.

On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.

Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.

Decision four: freshness without a retraining bill

Two mechanisms get conflated constantly, and the difference sets your monthly cost.

Automatic training creates a new solution version. The model learns from your latest data. It costs a training run each time.

Automatic updates are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.

So automatic updates solve the cold item problem and do nothing about model drift. AWS’s own guidance is that even with them, training cadence should stay at least weekly.

Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.

The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.

Decision five: how the recommendation reaches the page

Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.

Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.

Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.

Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.

Troubleshooting an AWS ecommerce recommendation pipeline

• Everyone gets the same recommendations. Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.
• New products never show up. Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.
• Training fails on data volume. Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.
• The dataset shrank and nothing errored. A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.
• Recommendations ignore what the user just clicked. Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.
• The bill rose during a quiet week. Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.
• Filters are not excluding purchased items. Filtering looks at a bounded slice of each user’s history per event type. Heavy users push older purchases outside it.

Auditing an account you did not build starts with listing what exists, not with reading the application code.

aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN

Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.

Common mistakes

• Leaving automatic training on for a solution that exists only to satisfy curiosity.
• Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.
• Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.
• Recording events only for authenticated users, starving the model of most of your traffic.
• Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.
• Calling PutEvents synchronously on the render path.
• Following an implementation guide that still references the retired HRNN recipes.
• Shipping with no fallback, so the first slow inference call becomes a slow product page.

Best practices

• Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.
• Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.
• Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.
• Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.
• Prefer batch inference plus a cache for anything that does not have to react within the session.
• Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.
• Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.

Frequently asked questions

How much data do I need before Personalize is worth it?

The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.

Can I build this without Amazon Personalize?

Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.

What does an idle Personalize setup cost?

More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.

Real-time or batch recommendations?

Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.

Do new products appear automatically?

Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.

How do I connect a Shopify or WooCommerce store?

There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.

How often should I retrain?

At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.

The one thing worth remembering

An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.

Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.

Need a second pair of eyes on your recommendation pipeline?

This is the kind of work I do for ecommerce teams building on AWS:

• Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.
• Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.
• Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.
• Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.
• Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.
• Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.

If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.

The post Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite appeared first on John Nessime.

That is the failure mode that makes this comparison worth writing about. The Parquet vs JSON vs CSV question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.

This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else’s benchmark, and an honest look at where CSV and JSON still beat the columnar option.

The billing model is the whole story

Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.

Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery’s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.

That last part catches people. LIMIT 10 does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.

So the question is not “which format is smaller on disk.” It is “how few bytes can the engine get away with reading.” Those diverge fast.

Four levers, not one

When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.

1. Column pruning

Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.

2. Block skipping

Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.

The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group’s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.

3. Splittability

This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.

Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.

If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.

4. Request overhead

Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine’s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.

CSV: still the right answer more often than people admit

Where it wins. Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.

Where it doesn’t. No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.

CSV is an excellent transport format and a poor storage format. That distinction is the useful one.

JSON: the most expensive way to store a table

Where it wins. Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.

Where it doesn’t. Every record repeats every key name. Store ten million events with a field called customer_identifier and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.

JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.

Parquet: cheap reads, real obligations

Where it wins. Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.

Where it doesn’t. This is the part the enthusiastic posts skip.

• You cannot append. Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.
• Schema mismatches across files are a runtime error, not a warning. Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.
• Timestamps are a compatibility minefield. The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.
• It is opaque. You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.
• Writing costs more. Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.

None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.

Measure it on your data, in about twenty minutes

Published benchmarks tell you about somebody else’s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.

1. Get a real byte count for what you have now. Not an estimate, the actual prefix.

aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable

The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.

2. Convert a representative slice, not the whole thing. One partition is enough. In Athena, a CTAS statement writes the output for you.

CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';

Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.

3. Run your three most common queries against both tables and compare bytes scanned. Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.
4. On BigQuery, price it before you run it. A dry run costs nothing and returns the byte estimate.

bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'

5. Inspect the Parquet file you just wrote. Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.

python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"

That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.

6. Put a guardrail up before anyone else touches it. Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.

Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.

Arguments that don’t survive contact

• “We gzip our CSVs, so we already save the money.” Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.
• “Parquet is always faster.” Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.
• “We’ll just convert everything to Parquet.” Converting a landing zone that receives appended events all day means fighting the format’s biggest weakness. Land raw, convert on a schedule, query the converted copy.
• “Storage is cheap so format doesn’t matter.” Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.
• “Our BI tool caches, so nobody re-scans.” Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.

How I’d decide

Not a scorecard. A short sequence of questions, in order.

• Is this data leaving your organisation? CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.
• Is it arriving continuously, one record at a time? Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.
• Is it read more than once, by more than one person, with filters? Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.
• Is the whole dataset smaller than a few hundred megabytes? Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.
• Are you fighting append semantics, concurrent writers, or schema evolution? You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.

The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.

Frequently asked questions

How much smaller is Parquet than CSV?

It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.

Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?

Less than you would think, because once data is loaded into a BigQuery native table it lives in Google’s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.

Is ORC better than Parquet?

They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet’s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.

Why did my query cost the same after I converted to Parquet?

Usually one of three things. You are running SELECT *, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.

Should I use Snappy, GZIP or ZSTD for Parquet?

Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.

Can I query CSV and Parquet in the same table?

Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.

What about the cost of the conversion job itself?

Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.

The one thing worth remembering

The Parquet vs JSON vs CSV decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.

Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.

Need help cutting a data lake bill?

Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:

• Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves
• Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses
• Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control
• Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges
• Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types
• Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end

If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.

The post Parquet vs JSON vs CSV: What Each Format Actually Costs You appeared first on John Nessime.

The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.

That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.

What managed compaction actually does

S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.

Compaction is the Iceberg rewriteDataFiles routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table’s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.

There are four strategies. auto is the default and picks for you: sort order defined in metadata means sort, otherwise binpack. binpack packs small files into bigger ones. sort orders rows hierarchically by your chosen columns so engines prune more files. z-order interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.

Both sort and z-order need a sort_order in your Iceberg table properties and the s3tables:GetTableData permission to set. They also cost more per byte processed than binpack, because they do more work.

Failure family one: maintenance that fails without telling anyone

This is the one worth reading the post for. Maintenance jobs have four states: Successful, Failed, Disabled, and Not_Yet_Run. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.

aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table

The response carries a block for each of icebergCompaction, icebergSnapshotManagement, and icebergUnreferencedFileRemoval, each with its own status, a lastRunTimestamp, and a failureMessage. The three are independent. Compaction reporting Successful tells you nothing about whether the other two are running.

The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.

Tags and branches disable snapshot expiry for the whole table

If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.

Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month’s bill rather than in their terminal.

Iceberg retention properties conflict with the managed config

If history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in metadata.json. It refuses instead.

This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.

ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');

gc.enabled set to false

Setting the Iceberg gc.enabled table property to false stops both snapshot management and unreferenced file removal. Leave it unset or explicitly true. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.

KMS key policy missing the maintenance principal

If the table is encrypted with a customer managed key, that key’s policy has to grant access to the service principal maintenance.s3tables.amazonaws.com. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.

Failure family two: the storage tail you keep paying for

Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.

When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from minSnapshotsToKeep (default 1) and maxSnapshotAgeHours (default 120). Both conditions must be satisfied before a snapshot goes.

When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for nonCurrentDays (default 10), then are deleted permanently.

Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through unreferencedDays (default 3) into the same noncurrent window.

Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.

# Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'

Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.

Failure family three: write amplification and the compaction charge

S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.

The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.

People reach for targetFileSizeMB first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.

The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.

Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.

Failure family four: files compaction quietly skips

Compaction supports Parquet, Avro, and ORC. It does not support the Fixed data type, nor the brotli or lz4 compression codecs.

That last one matters because lz4 is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.

Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the write.format.default table property to avro or orc. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.

Failure family five: compaction and your writers fighting over the table

Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.

When compaction loses you see Failed with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between Failed and Successful.

When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.

Turning managed compaction off, and when that is right

Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.

The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.

Disabling is a per-table change, and it does not require zeroing the settings.

aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'

From there you run maintenance yourself. Trino exposes an optimize procedure through ALTER TABLE ... EXECUTE, and Spark has the standard Iceberg rewrite_data_files procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.

Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.

Troubleshooting S3 Tables compaction and maintenance

Work through these in order. The first step resolves more cases than the rest combined.

1. Check all three job statuses, not one. Run get-table-maintenance-job-status and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.
2. Read the failure message literally. It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means history.expire.* is set.
3. Status is Disabled or Not_Yet_Run unexpectedly. Someone replaced the configuration. Confirm with get-table-maintenance-configuration, and remember put-table-maintenance-configuration replaces rather than merges.
4. Compaction runs but file count stays high. Check codec and data types against the unsupported list. Files using brotli or lz4, or containing the Fixed type, are outside what compaction touches.
5. Insufficient-access failures on an encrypted table. Check the KMS key policy grants the maintenance.s3tables.amazonaws.com service principal, and that the key is enabled and still exists.
6. Storage still growing with everything green. Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.

For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer “when did this start” rather than “what is broken now”.

Common mistakes

• Treating the three maintenance jobs as one thing. They fail independently and for different reasons. “Maintenance is enabled” is not a status.
• Creating branches or tags on a managed table. A normal Iceberg workflow that switches off snapshot expiry for the whole table.
• Tuning targetFileSizeMB to reduce cost. It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.
• Setting sort or z-order without a sort order defined. The strategy needs sort_order in table properties and the s3tables:GetTableData permission, and it costs more per byte than binpack.
• Running write pipelines with no commit retry. Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.

Best practices for running S3 Tables compaction in production

• Poll job status on a schedule and alert on it. A small scheduled function calling get-table-maintenance-job-status across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on Failed and on a stale lastRunTimestamp, on whatever dashboard you already watch, Grafana or CloudWatch.
• Make maintenance config part of your IaC. These are settings someone can change with one CLI call and no review. Codify them so drift is visible.
• Set retention windows from a real requirement. Ask how far back anyone needs to time travel, then set maxSnapshotAgeHours and nonCurrentDays to that plus a margin, rather than accepting defaults by omission.
• Batch on the write side before tuning anything else. Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.
• Choose the strategy from the query pattern. One dominant filter column suggests sort. Several equally weighted predicates suggest z-order. Neither suggests leaving it on auto.
• Track the compaction billing dimension separately. It moves independently of storage and it is the earliest signal that a writer’s behaviour has changed.

Frequently asked questions

How do I know if S3 Tables compaction is actually running?

Call get-table-maintenance-job-status and read the icebergCompaction block. It returns Successful, Failed, Disabled, or Not_Yet_Run, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.

Why is my S3 Tables storage growing when compaction says it succeeded?

Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.

Can I run my own compaction alongside the managed one?

You can, but it is a poor idea. Both are writers competing under Iceberg’s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.

Does disabling compaction stop all S3 Tables charges?

No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.

Is sort or z-order compaction worth the extra cost?

It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour sort. Several dimensions in varying combinations favour z-order. Queries that mostly scan everything justify neither.

Can I recover data deleted by S3 Tables maintenance?

Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.

The one thing worth remembering

S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.

What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on get-table-maintenance-job-status that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.

Need help with Iceberg tables on AWS?

I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:

• Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots
• Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana
• Working out where a bucket’s storage growth actually comes from, and setting retention windows against a real requirement
• Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them
• Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick
• Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance

If something is not adding up, send me the output of get-table-maintenance-job-status for the table and a rough description of your write pattern. That is usually enough to say what is going on.

The post S3 Tables Compaction in Production: The Cost of Convenience appeared first on John Nessime.

That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.

So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It’s three services and a weekend, and it works on the first try, which is exactly the problem. A secure patient document portal built this way passes every test you think to run, because the tests you think to run are all “can Alice download Alice’s file?” The one that matters is “what can Alice see about Bob?” and that one has an answer you won’t like.

This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can’t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.

The failure that never pages you

Here’s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*. You test it. Alice gets her file. Alice cannot get Bob’s file. Ship it.

Then, because the front end needs to render a file list, someone adds s3:ListBucket. And s3:ListBucket is not an object-level action. It applies to the bucket ARN, not the object ARN, so the ${cognito-identity.amazonaws.com:sub} substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.

They still can’t download anything. Every GetObject outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. patients/8f2a.../oncology-referral-2024.pdf tells you something about a person even if the bytes stay locked.

Nothing alerts on this. No error appears in any log you’re watching, because nothing errored. The listing succeeded. That’s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn’t announce itself.

The fix is a condition key on the list action, not a resource string:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}

Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you’ve secured the download path while leaving the discovery path wide open.

One trap in that variable. ${cognito-identity.amazonaws.com:sub} resolves to the identity ID from the Cognito identity pool, not the sub claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that’s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.

Decide where the authorization boundary lives

There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.

Pattern A: browser holds AWS credentials

The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.

It’s less code and there’s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it’s harder to reason about than application code because the enforcement happens in a service you can’t step through with a debugger.

The real limitation is expressiveness. IAM policy conditions do not know that Bob’s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.

Pattern B: backend brokers every object

The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API’s execution role.

You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.

For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (“can we hide results until the doctor reviews them?”) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.

If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.

Your download links expire before you think they do

This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.

A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default AssumeRole session is one hour.

Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That’s the source of the flapping. You can watch it yourself:

# Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.

The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone’s inbox.

Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.

Encryption that actually gates something

New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence “data is encrypted at rest” and gates precisely nothing, because anyone with s3:GetObject gets plaintext back. It is a storage property, not an access control.

Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and kms:Decrypt on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.

You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn’t think about the other.

The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I’d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.

The audit trail you don’t have yet

Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don’t know.

CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are data events, and data events are off by default because of volume and cost. GetObject on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.

# Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'

S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is “who.”

Two things to sort out while you’re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal’s roles cannot write to, so that compromising the application doesn’t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.

Cutting off access when someone leaves

Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.

Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it’s a checkbox someone ticked to stop users complaining about re-login.

Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. AdminUserGlobalSignOut revokes all of a user’s refresh, ID, and access tokens from the server side, and the /oauth2/revoke endpoint kills one refresh token and everything descended from it.

# Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'

Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.

Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn’t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you’re paying for Plus for threat protection features, you already have it.

Troubleshooting the usual suspects

Every request returns access denied after login

Nine times out of ten the prefix in your policy variable doesn’t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.

Signature does not match on a presigned URL

Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the X-Amz-SignedHeaders list in the URL against what the client actually sent.

Uploads succeed but downloads fail

Check the KMS key policy separately from the IAM policy. A role can hold kms:GenerateDataKey (enough to write) without kms:Decrypt (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.

Browser uploads fail before any request reaches S3

That’s CORS. The bucket needs a CORS configuration allowing your portal’s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.

Nothing appears in CloudTrail

Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.

Common mistakes

• Granting s3:ListBucket without an s3:prefix condition, so patients can enumerate every key even though downloads are blocked.
• Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.
• Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.
• Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.
• Assuming default encryption means access control. It means the disk is encrypted, nothing more.
• Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.
• Skipping CloudTrail data events to save money, then discovering the gap during an access review.

Best practices for a secure patient document portal

• Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.
• Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.
• Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.
• Use a KMS customer managed key so decrypt authority is separable from S3 authority.
• Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.
• Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.
• Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.
• Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that’s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.
• Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.

Frequently asked questions

Are S3 and Cognito enough for HIPAA compliance?

Both are covered under AWS’s HIPAA eligibility program, and Cognito’s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.

Should I use a Cognito identity pool or a backend API?

Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.

How long should a presigned URL last?

Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.

Can patients upload documents back to the portal?

Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.

Does enabling CloudTrail data events cost much?

Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.

What happens to sessions when I disable a user?

Nothing immediate. Existing access tokens remain valid until they expire. Call AdminUserGlobalSignOut to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.

Do I need CloudFront in front of the bucket?

Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.

The one thing to remember

A signed-in token proves who someone is. It does not decide what they may reach. In a secure patient document portal the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.

The demo passes either way. The difference only shows up when someone reads a key name they shouldn’t have seen, and nothing in your system considers that an error.

If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you’ve found your next task.

Need a second pair of eyes on your portal?

This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.

• Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.
• Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.
• Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.
• Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.
• Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.
• Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.

Send me your bucket policy and your identity pool role, with the account IDs redacted, and I’ll tell you what a signed-in patient can actually see.

The post Building a Secure Patient Document Portal on S3 and Cognito appeared first on John Nessime.

That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution’s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.

This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.

What “personalized” has to mean before you build anything

Personalization in an education setting is three separate questions that people tend to collapse into one:

• What is this learner allowed to see? Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.
• What has this learner already done? Submissions, attempts, viewed material. This is state, and it belongs in a database you control.
• How is this learner doing? Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.

Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying “do not reveal solutions before the due date” is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.

Where a personalized learning assistant on AWS actually breaks

Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.

Failure family one: identity that stops at the front door

The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool’s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform’s published keys.

Almost everyone gets that far. The failure is what happens next.

The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.

Take the scope from the launch, then keep it server-side

The pattern that holds up:

1. Validate the launch JWT against the platform’s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.
2. Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.
3. Derive every retrieval filter from that session record. The client never names a course, a section or a role.
4. If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.
5. If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.

Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.

Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.

Failure family two: retrieval that ignores the boundary

This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.

Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.

The sidecar file is the only metadata that counts

For an S3 data source, attributes come from a companion file sitting next to the document with .metadata.json appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.

{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}

Note visibility as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get visibility: "instructor", and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.

At query time you compose the filter server-side from the session record. Bedrock’s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:

{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}

Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk’s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.

Silo, pool or bridge

AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:

• Silo gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.
• Pool puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.
• Bridge splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.

For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district’s data protection officer goes very differently when your answer is “we filter” rather than “they are separate indexes.”

Failure family three: guardrails that do less than you assume

Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.

Contextual grounding does not cover chatbots. AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is “we turned on grounding checks,” read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.

PII detection is probabilistic and text-only. The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.

Blocked content lands in your logs. AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.

None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.

Failure family four: the cost shape of an academic term

Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.

The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.

Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:

• Aurora PostgreSQL with pgvector. A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.
• Amazon S3 Vectors. Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.
• OpenSearch Serverless. The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.

On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.

Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.

Failure family five: no evidence when someone asks

FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor’s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.

What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:

• Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.
• Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.
• Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.
• Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.
• Write down retention for conversation history and enforce it with lifecycle rules. “We keep it forever because storage is cheap” is a defensible engineering position and an indefensible one in an education procurement review.
• Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.

None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution’s legal team gets involved.

Troubleshooting the things that actually page you

“The new material isn’t showing up”

Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.

aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID

Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.

“The filter returns nothing at all”

Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. BIO201 and bio201 are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.

“The launch shows a blank iframe”

Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.

“Answers are right but the citations are wrong”

Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.

“Grades aren’t reaching the gradebook”

Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.

Common mistakes

• Enforcing access control in the system prompt. Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.
• Trusting client-supplied course or role values. Derive scope from the validated launch token and store it server-side, always.
• Ingesting the whole course folder. Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.
• Putting real names and student numbers in prompts. Pseudonymize before the model call and resolve afterwards.
• Sizing the vector store from peak week. The bill is set by the idle floor, and in education idle is most of the year.
• Assuming a guardrail feature does what its name implies. Read the supported use cases. Contextual grounding on a chatbot is the classic example.
• Piloting with one enthusiastic lecturer. That lecturer’s material is unusually well organized. The second department will not be.

Best practices

• Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.
• Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.
• Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.
• Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.
• Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.
• Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.
• Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.

Frequently asked questions

Do we need to fine-tune a model for our curriculum?

Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.

How do we stop it from just doing the homework?

Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.

Is Bedrock the right choice versus calling a model API directly?

If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.

Does student data get used to train the model?

AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.

What does it cost to run a personalized learning assistant on AWS?

Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.

How do we know whether it actually helps anyone learn?

Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant’s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.

The one thing worth remembering

A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.

Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.

Need a hand building or auditing one of these?

I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:

• Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.
• Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.
• Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.
• Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.
• Cost allocation tagging by course and faculty, with dashboards that answer “which department is spending this” without handing out AWS console access.
• Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.

If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.

The post Building a Personalized Learning Assistant on AWS Without Leaking Student Data appeared first on John Nessime.

That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.

The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.

What this post covers

Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn’t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.

Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.

Failure family one: there is no join key, only a filename

Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.

Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.

What to use instead

Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.

Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for “this exact bytes-on-disk artefact.” If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.

S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.

Failure family two: extraction confidence gets flattened into a fact

Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.

Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn’t go away. You just stopped being able to see it.

Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always “which extractor produced these,” and without that column the answer is a week of archaeology.

-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);

The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.

Failure family three: two catalogs, two permission models

This is the one that bites teams who did everything else right.

Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.

The result is predictable and unpleasant. An analyst who is correctly denied the salary column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.

There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:

• Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.
• Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.
• Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.
• Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.

Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with .metadata.json appended. That file is where classification, tenant, document ID and effective dates belong.

{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}

Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.

Failure family four: index decisions you cannot take back

S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.

By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.

If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.

# Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'

The dimension has to match your embedding model’s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.

Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.

Failure family five: reprocessing rewrites history

You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter’s board pack change.

This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.

Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and “why did this number move” has an answer you can query instead of guess at.

Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.

A shape that works for combining structured and unstructured data on AWS

Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.

1. Land raw, enable versioning, enable S3 Metadata. Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.
2. Mint the document ID at landing. Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.
3. Classify before you extract. Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.
4. Extract to a staging table, not straight to the curated table. Staging gives you somewhere to run validation before anything touches a table an analyst can query.
5. Merge into Iceberg on document identity, carrying confidence and provenance columns. Never merge on object key.
6. Embed separately, keyed by the same document ID. The vector plane and the table plane are different systems with different refresh rates. Treat them as such.
7. Join at query time in Athena. Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.

Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.

Troubleshooting

Athena says the principal has no privilege on the resource

This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.

Queries against metadata tables fail with a syntax error

Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.

Knowledge base sync fails partway with validation errors

Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.

Retrieval returns documents the caller should not see

Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.

Athena gets slower week over week on a table that isn’t growing much

Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.

Common mistakes

• Joining on the S3 object key because it was there and it worked in the prototype.
• Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.
• Applying a confidence threshold and discarding the score.
• Creating the vector index before deciding on the embedding model.
• Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.
• Reprocessing in place, with no record of which extractor produced which value.
• Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.

Best practices

• One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.
• Confidence and provenance columns are not optional. They are the difference between debugging and guessing.
• Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.
• Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.
• Enforce classification in all three planes independently. There is no single choke point.
• Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.
• Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.

FAQ

Do I need a lakehouse to combine structured and unstructured data on AWS?

No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.

Should extraction output live in the same table as the structured source data?

Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.

Is S3 Vectors a replacement for OpenSearch?

For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.

How do I keep the vector index in sync with the documents?

Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because “mostly in sync” degrades quietly.

What breaks first when this architecture is done badly?

Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.

Can I use Textract instead of Bedrock Data Automation?

Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.

Does Lake Formation govern my documents as well as my tables?

It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.

The one thing worth remembering

Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.

Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.

Need help wiring documents and tables together?

This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:

• Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.
• Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.
• Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.
• Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.
• Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.
• Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.

Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.

The post The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS appeared first on John Nessime.

Nothing. Not a cent. Oracle counts a vCPU on Amazon RDS exactly the way it counts a vCPU on Amazon EC2, and it says so in the same paragraph of the same policy document. The managed service does not buy you a discount on the software.

That single fact reshapes the whole argument. Once the license line is identical on both sides of the page, Oracle on EC2 vs RDS stops being a cost comparison and becomes a much more interesting question: which set of constraints can your application actually live inside? This post walks through the real differences, the places each option quietly hurts, the middle option that is being retired, and a decision procedure you can run before anyone opens a pricing calculator.

Start with the thing that is not a difference

Oracle publishes a policy called “Licensing Oracle Software in the Cloud Computing Environment.” It names a short list of Authorized Cloud Environments, and both Amazon EC2 and Amazon RDS are on it, alongside Microsoft Azure and Google Cloud Platform. Inside an Authorized Cloud Environment the counting rules are these:

• Two vCPUs count as one Oracle Processor license when multi-threading of processor cores is enabled.
• One vCPU counts as one Processor license when multi-threading is not enabled.
• The Oracle Processor Core Factor Table does not apply. The 0.5 multiplier you rely on for x86 hardware in your own data center is simply gone.
• For Standard Edition products, instances with four or fewer vCPUs count as one socket. Above four, every four vCPUs rounded up to the nearest multiple of four equals one socket.
• Standard Edition 2 may only be licensed on instances up to eight AWS vCPUs. That is a hard ceiling, and it is the single most common reason an SE2 estate has to become an Enterprise Edition estate on migration.

None of those rules distinguish EC2 from RDS. The policy treats them as the same environment for counting purposes.

There is one genuine licensing difference, and it is narrow. RDS offers a License Included model where AWS holds the Oracle license and bundles it into the instance-hour rate. That model covers Standard Edition 2 only. Enterprise Edition on RDS is bring-your-own-license, full stop. On EC2 there is no License Included option at all, so every Oracle database you run there is BYOL by definition.

Worth reading the footer of that Oracle policy document, too. It states that the document is for educational purposes and may not be incorporated into any contract. Your Oracle Master Agreement and ordering documents are what actually bind you. The cloud policy is how Oracle says it will behave, not what it promised in writing, and it can change without notice. Pin a copy of the version in force when you deploy and file it with your architecture decision record.

Find out what you are actually using first

Before you compare anything, find out which licensable options and management packs your current database is touching. Oracle tracks this itself. The feature usage view records every separately licensed feature the database has detected in use, which is exactly what an auditor looks at.

-- Which separately licensed features has this database touched?
SELECT name,
       detected_usages,
       currently_used,
       first_usage_date,
       last_usage_date
FROM   dba_feature_usage_statistics
WHERE  detected_usages > 0
ORDER  BY name;

Oracle also publishes an options and packs usage script through My Oracle Support that goes further and correlates usage across several dictionary views. If you have a support contract, use that one rather than rolling your own. The output frequently surprises people. Partitioning, Advanced Compression, and the Diagnostics and Tuning Packs get switched on by a well-meaning DBA or by a default setting and then sit there generating an entitlement gap.

Amazon RDS for Oracle: where it wins and where it hurts

Where RDS wins

• The operational floor is done for you. Automated backups with point-in-time recovery, minor version patching in a maintenance window, Multi-AZ synchronous standby with automatic failover, storage autoscaling. None of that needs a person on a rota.
• License Included removes the Oracle relationship entirely. For an SE2 workload where you own no licenses, this is genuinely hard to beat. There is no entitlement to track, no support renewal to chase, and no Oracle audit surface for the database software.
• Read replicas are configured for you. RDS manages the Data Guard setup behind cross-region replicas, including the network path between primary and standby. If the replica runs open read-only, that is Active Data Guard and you need the license for it, but the plumbing is not your problem.
• Parameter groups make configuration reviewable. Database parameters live in a versioned object you can diff, rather than in a spfile someone edited at two in the morning.

Where RDS hurts

The master user on an RDS for Oracle instance gets the DBA role with pieces removed. AWS documents exactly which privileges are withheld to keep the managed experience intact:

• ALTER DATABASE
• ALTER SYSTEM
• CREATE ANY DIRECTORY and DROP ANY DIRECTORY
• GRANT ANY PRIVILEGE and GRANT ANY ROLE

You also cannot log in as SYS, SYSTEM, or any other Oracle-supplied administrative account. This is the wall most migrations hit, and they hit it late. A packaged vendor application ships an installer that expects to connect as sysdba, create a directory object pointing at a local path, and run ALTER SYSTEM a dozen times. On RDS that installer does not complete. There is no workaround, because the privilege is not withheld by policy, it is withheld by design.

Three more constraints that bite later rather than sooner:

• Block size is fixed at creation. RDS for Oracle instances are created with an 8 KB default block size and it cannot be changed afterwards. The SYSTEM and temporary tablespaces always use it. You can create additional tablespaces at other block sizes by allocating the matching cache parameter and using the BLOCKSIZE clause, but the core of the database is fixed.
• Single files cap at 16 TiB. That is the ext4 filesystem limit on the underlying host, so bigfile data files inherit it. Try to grow past it and you get an ORA-01237 with a “File too large” error underneath.
• No host access at all. No shell, no third-party backup agent, no monitoring agent installed on the database server, no custom scripts on the filesystem. If your compliance tooling or your APM vendor requires an on-host agent, that requirement decides the platform for you.

Oracle on EC2: where it wins and where it hurts

Where EC2 wins

• Full sysdba and full root. Everything the vendor installer wants is available. Directory objects, external tables reading from local paths, custom PL/SQL that shells out, agents, whatever the packaged application assumed about its host.
• The full replication toolbox. Data Guard, Active Data Guard, and GoldenGate all run natively on EC2, and you control the protection mode, the apply lag tolerance, and the failover behaviour rather than accepting a managed default.
• Instance shape is a license lever. This is the underrated one. Oracle bills you on vCPUs, and most Oracle databases are memory bound rather than thread bound. On EC2 you can pick a memory-dense family and hit your SGA target without buying vCPUs, and therefore licenses, that the workload never uses. RDS gives you a narrower catalogue to choose from.
• You choose when to patch. No maintenance window imposed from outside, no forced minor version upgrade, no surprise reboot on a night your business cannot take one.
• Storage is yours to shape. Separate EBS volumes for data, redo, and archive logs, with independent IOPS and throughput. RDS abstracts that away.

Where EC2 hurts

Every one of those wins is a job someone now has to do, forever. The honest cost of Oracle on EC2 is not the instance-hour, it is the rota.

• Backups you have to prove. RMAN to S3 is a solved pattern, but a backup that has never been restored is a hypothesis. Schedule a real restore into a throwaway instance on a cadence and treat a failed restore as an incident.
• A standby that drifts silently. Data Guard apply lag is the classic invisible failure. Everything looks green, the standby is mounted, redo is shipping, and apply stopped three days ago because of a single nologging operation or a filesystem that filled. You find out during the failover.
• Patching is now a project. Quarterly Oracle patches, OS patches, and the interaction between them. On a Data Guard pair that means a standby-first sequence and a rollback plan.
• Monitoring you have to build. Tablespace growth, apply lag, wait events, ASM disk group headroom, archive log destination fullness. Amazon CloudWatch gets you host metrics and nothing about the database. Teams typically land on a Prometheus exporter feeding Grafana, either self-hosted or through Grafana Cloud or Amazon Managed Grafana, plus alerting that someone actually answers.
• Cost attribution gets murky. Instance, EBS, snapshots, cross-AZ traffic, and the license amortisation all land in different places. A cost visibility tool such as Vantage or CloudZero earns its keep here, because the Oracle line is the one finance will ask about.

The middle option that is being retired

For a while the answer to “I need sysdba but I also want managed backups” was Amazon RDS Custom for Oracle. It gave you shell access to the underlying EC2 instance and sysdba on the database, while keeping automated backups and host recovery, policed by a mechanism called the support perimeter that flagged your instance as unsupported if a change broke the automation.

AWS has announced that support for RDS Custom for Oracle ends on 31 March 2027. After that date the console and the resources go away, including instances, snapshots, and custom engine versions. AWS’s own guidance is to move those workloads to self-managed Oracle on EC2.

This is one of the few dates worth putting in an article, because it removes an option from your decision tree. If you are choosing a platform today, RDS Custom for Oracle is not a candidate. If you are already on it, the migration is on your roadmap whether you have written it down or not. Note that this retirement does not touch RDS for Oracle or RDS Custom for SQL Server.

The cost levers that actually move the number

Once you accept that the platform choice does not change the license rate, these are the things that do change the bill.

1. Memory-to-vCPU ratio. Reaching a memory target by moving up within a general purpose family buys vCPUs you did not need, and each pair of them is another Processor license. Picking a memory-optimised family instead is often the largest single saving available.
2. Multi-AZ under BYOL doubles the count. AWS states this plainly in its own documentation: under BYOL you must have a license for both the primary and the standby DB instance in a Multi-AZ deployment. Enabling the checkbox on a BYOL instance without buying the second set of licenses puts you out of compliance the moment it comes up.
3. Options and management packs. These are priced per Processor on top of the database license. On RDS, option groups can put some of them one click away, which is convenient and expensive in equal measure.
4. Edition, forced by the vCPU ceiling. SE2 stops at eight vCPUs in an Authorized Cloud Environment. A workload that needs more compute has to move to Enterprise Edition, and that is a different order of magnitude, not a percentage increase.
5. Infrastructure commitments. Reserved Instances and Savings Plans reduce the AWS side of the bill on both platforms. They do nothing to the Oracle side.

Two quick checks worth running before you model anything. First, confirm what an instance is actually presenting, because the multi-threading state is what decides whether you divide by two:

# On an EC2 instance: threads per core tells you the multi-threading state.
# "Thread(s) per core: 2" means the 2 vCPU = 1 Processor rule applies.
lscpu | grep -E 'Thread(s) per core|Core(s) per socket|^CPU(s):'

Second, check what your existing RDS instances are declaring, since the license model and the Multi-AZ flag together determine your exposure:

# Every Oracle RDS instance with its class, license model and Multi-AZ state.
aws rds describe-db-instances 
  --query "DBInstances[?starts_with(Engine, 'oracle')].
[DBInstanceIdentifier,Engine,DBInstanceClass,LicenseModel,MultiAZ]" 
  --output table

A MultiAZ value of true next to a LicenseModel of bring-your-own-license is the line to look at twice.

How I would decide between Oracle on EC2 vs RDS

Run these in order. The first question that returns a hard “no” makes the decision for you, and you can stop.

1. Does anything need sysdba or host access? Vendor installers, on-host agents, directory objects pointing at local paths, external tables reading local files, custom OS-level jobs. If yes, it is EC2. Nothing else in this list matters.
2. Run the feature usage query. Establish which options and packs you are on the hook for today. That number travels with you regardless of platform, and it is usually larger than the team expects.
3. Check the physical constraints. Non-8 KB block size on the SYSTEM tablespace, or a single data file heading past 16 TiB, both rule RDS out.
4. Settle the edition. If SE2 fits inside eight vCPUs and you own no perpetual licenses, RDS with License Included is very hard to argue against. If you need Enterprise Edition, you are BYOL on either platform and the licensing argument is over.
5. Count licenses at the shape you would really run. Production instance, plus standby if you want Multi-AZ or Data Guard, plus every option pack from step two. Do this before you look at instance-hour rates, because it is the bigger number.
6. Price the rota honestly. Who runs restore tests, who owns the patch calendar, who gets paged at three in the morning when apply lag alerts. If the answer is “we will figure it out,” that is a vote for RDS.
7. Pick, then write down why. Record the Oracle policy version you counted against and the instance shapes you assumed. When an audit letter arrives, that document is the difference between a conversation and a negotiation.

One useful side effect of building a dev or reporting copy outside AWS entirely: a modest VPS from a provider like Contabo or InterServer running Oracle’s free Express Edition gives developers a schema to work against without touching a licensed instance. It will not stand in for production, but it takes pressure off the environments that cost money.

Arguments that do not survive contact

• “RDS is cheaper because AWS manages it.” Under BYOL the license line is identical. What you save is operational labour, which is a real saving but a different budget line, and you should say which one you mean.
• “We will disable multi-threading to reduce licenses.” Turning it off halves your usable threads and flips the ratio to one vCPU per Processor license. You end up at the same license count with less compute.
• “The core factor gives us 0.5 on x86.” Not in an Authorized Cloud Environment. Oracle states the table does not apply there. This is the source of the widely repeated claim that moving to the cloud doubles Oracle licensing on identical hardware.
• “The standby is passive, so it is free.” AWS documents that BYOL Multi-AZ requires licensing both instances. There is a separate Oracle provision about genuinely cold failover nodes, but a synchronous standby that is always running is not that. Get your specific position confirmed in writing by whoever advises you on Oracle contracts, not from a blog, including this one.
• “Graviton is a different architecture, so it licenses differently.” The cloud policy counts vCPUs. It does not carve out processor architectures on shared tenancy.
• “We will start on RDS and move to EC2 if it does not work.” Technically true and practically expensive. Moving off RDS means a full migration project with a cutover window, not a configuration change. Establish the sysdba requirement before you build, not after.

Frequently asked questions

Is Oracle licensing cheaper on RDS than on EC2?

Under bring-your-own-license, no. Oracle’s cloud policy names EC2 and RDS together and applies the same vCPU counting to both. The only structural difference is that RDS offers a License Included model, which is available for Standard Edition 2 only.

Can I run Oracle Enterprise Edition with License Included on RDS?

No. AWS supports License Included for Standard Edition 2 only. Enterprise Edition on RDS is BYOL, and you need the appropriate licenses with active Software Update License and Support.

Do I need to license the Multi-AZ standby?

Under BYOL, AWS documentation states you must have a license for both the primary and the standby DB instance. Budget for double the Processor count before you enable it. Under License Included the standby is reflected in the instance-hour rate instead.

What is the largest Standard Edition 2 instance I can run on AWS?

Oracle’s cloud policy caps Standard Edition 2 at eight AWS vCPUs. Past that you are into Enterprise Edition. This ceiling catches a lot of migrations that were sized on memory requirements rather than vCPU count.

Can I get sysdba access on RDS for Oracle?

No. The master user receives the DBA role minus a documented set of privileges including ALTER SYSTEM, ALTER DATABASE, CREATE ANY DIRECTORY, and the GRANT ANY privileges, and the Oracle-supplied administrative accounts are not available for login. If an application requires sysdba, EC2 is your platform.

Is RDS Custom for Oracle still a viable middle ground?

Not for new work. AWS ends support for RDS Custom for Oracle on 31 March 2027 and recommends moving those workloads to EC2. Existing users should treat the migration as a scheduled project rather than a future problem.

Does turning off hyperthreading reduce my Oracle license count?

No. With multi-threading enabled the ratio is two vCPUs per Processor license. With it disabled the ratio becomes one to one. The license requirement stays where it was while your available compute halves.

Conclusion

If you remember one thing from the Oracle on EC2 vs RDS comparison, make it this: the platform choice does not change what Oracle charges you. It changes who does the work and what your application is allowed to do.

So run the decision in that order. Establish the sysdba requirement first, because it is binary and it ends the discussion. Count your options and packs second, because that number is usually wrong in the direction that costs money. Compare instance-hour rates last, because it is the smallest term in the equation and the one everyone starts with.

And write down which version of Oracle’s cloud policy you counted against. It is a non-contractual document that can change, and the copy you filed on the day you deployed is worth more than any argument you can make from memory later.

Need a second pair of eyes on an Oracle migration to AWS?

Most of the expensive mistakes here happen before anything is built, in the assumptions nobody wrote down. Things I can help with:

• Auditing feature and option pack usage on your existing databases so you know your real entitlement position before you size anything
• Working through the sysdba and host-access requirements of packaged applications to determine whether RDS is even on the table
• Modelling Processor counts across candidate instance shapes, including standby and Multi-AZ exposure, so the number in the business case survives review
• Building the operational layer that self-managed Oracle on EC2 needs: RMAN to S3, tested restores, Data Guard apply-lag alerting, and patch runbooks
• Planning migrations off RDS Custom for Oracle to EC2 ahead of the end-of-support date
• Setting up monitoring and cost attribution that separates the AWS line from the Oracle line, so finance gets an answer instead of a total

Send me the output of your feature usage query, or a list of the instance classes you are considering, and I will tell you what I would check first.

The post Oracle on EC2 vs RDS: Why the License Math Doesn’t Save You appeared first on John Nessime.