That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.

That is the shape of every serious problem in this space. A personalized learning assistant on AWS almost never fails loudly. It fails by being helpful to the wrong person, or by being confidently wrong in a way that reads as authoritative because it came from the institution’s own system. Both failures are silent until a human reports them, and by then the assistant has been answering that way for a term.

This post covers the parts that are genuinely hard: getting identity and scope out of the learning platform instead of the browser, making retrieval respect the boundary, knowing what Bedrock Guardrails will and will not do for you, sizing a system whose traffic collapses every summer, and proving after the fact what the assistant saw. I assume you can already stand up a RAG pipeline. The pipeline is the easy part.

What “personalized” has to mean before you build anything

Personalization in an education setting is three separate questions that people tend to collapse into one:

• What is this learner allowed to see? Course enrollment, section, release dates, role. This is an authorization question and it belongs nowhere near the model.
• What has this learner already done? Submissions, attempts, viewed material. This is state, and it belongs in a database you control.
• How is this learner doing? Mastery, misconceptions, pace. This is inference, and it is the only part where a model earns its keep.

Teams that skip straight to the third question end up asking the model to enforce the first one through prompt instructions. That does not hold. A system prompt saying “do not reveal solutions before the due date” is a suggestion; a retrieval filter that never returns the solution chunk is a control. Keep the boundary in the retrieval layer and the personalization in the generation layer, and most of the scary failure modes stop being possible rather than becoming unlikely.

Where a personalized learning assistant on AWS actually breaks

Organized by failure family rather than by service, because the services are documented and the failures are not. Five of them, roughly in the order they bite.

Failure family one: identity that stops at the front door

The assistant lives inside Canvas, Moodle, Blackboard, Brightspace or Open edX. It gets there through LTI 1.3, which replaced the old shared-secret model with an OpenID Connect launch and an RSA-signed JWT. The platform initiates a login against your tool’s login endpoint, your tool responds with an auth request, the platform posts back a signed token, and you validate it against the platform’s published keys.

Almost everyone gets that far. The failure is what happens next.

The launch token carries a stable subject identifier, a roles claim, a context claim describing the course, and a deployment identifier, all namespaced under the 1EdTech spec URIs. Teams validate the signature, pull the user ID out, then build the rest of the session from whatever the front end sends afterwards. Course ID as a query parameter. Role in local storage. It works in testing because the front end sends the truth, and it stops working the moment a curious student opens developer tools. If your filter values arrive from the client, you have built an access control system that asks the attacker what they should be allowed to see.

Take the scope from the launch, then keep it server-side

The pattern that holds up:

1. Validate the launch JWT against the platform’s key set. Check the issuer, the audience, the nonce and the deployment identifier, not just the signature.
2. Extract subject, roles and context from the token and write them into a server-side session. Hand the browser an opaque session cookie and nothing else.
3. Derive every retrieval filter from that session record. The client never names a course, a section or a role.
4. If you need the full roster rather than just the launching user, call Names and Role Provisioning Services with an OAuth 2.0 client credentials token. That gives you learners who have never launched the tool, which matters for cohort-level features.
5. If you write grades back, do it through Assignment and Grade Services against a line item you created. A launch that works tells you nothing about whether passback works.

Test launch, roster fetch and grade passback as three independent contracts. They fail independently and they fail quietly. A term can start with a tool that launches beautifully and drops every grade on the floor because AGS was never wired to a line item.

Worth knowing before you commit: LTI 1.3 is meaningfully more setup than 1.1. Key registration, the OIDC handshake, per-deployment configuration on every platform instance. That is the price of not signing student traffic with a shared secret, and it is worth paying, but budget for it honestly. If you need to test against several platforms, a self-managed Moodle or Open edX instance on a cheap VPS from somewhere like Contabo or InterServer beats waiting on sandbox tenants, and you get to break it freely.

Failure family two: retrieval that ignores the boundary

This is the one from the opening. Vector search does not know about courses. It knows about similarity. If the answer key is semantically close to the question, it comes back, and the model uses it, and the response is fluent and correct and completely inappropriate.

Amazon Bedrock Knowledge Bases handles this through metadata filtering: you attach attributes to each document at ingestion, then pass a filter at query time that pre-filters the vector store before the similarity search runs. The filter is the control. Everything else is decoration.

The sidecar file is the only metadata that counts

For an S3 data source, attributes come from a companion file sitting next to the document with .metadata.json appended to the object key. This trips people up constantly, because S3 objects already have user-defined metadata and object tags, and neither of those is read for retrieval filtering. Keep lifecycle and cost allocation in S3 tags where they belong, and put filterable attributes in the sidecar.

{
  "metadataAttributes": {
    "course_id": "BIO201",
    "section_id": "BIO201-B",
    "institution_id": "uni-main",
    "material_type": "lecture_notes",
    "visibility": "student"
  }
}

Note visibility as a first-class attribute rather than an inference from folder structure. Rubrics, model answers and marking schemes get visibility: "instructor", and the student-facing filter excludes them by value, not by path convention. Path conventions get violated the first time a busy tutor uploads a file to the wrong folder.

At query time you compose the filter server-side from the session record. Bedrock’s retrieval configuration supports boolean composition, so the tenant condition and the visibility condition are separate clauses that must both hold:

{
  "vectorSearchConfiguration": {
    "filter": {
      "andAll": [
        { "equals": { "key": "institution_id", "value": "uni-main" } },
        { "equals": { "key": "course_id",      "value": "BIO201"   } },
        { "equals": { "key": "visibility",     "value": "student"  } }
      ]
    }
  }
}

Two habits make this survive production. Build the filter through a single function that always prepends institution and course, so no future endpoint can forget it. Then write a test that queries a known instructor-only chunk’s exact wording from a student session and asserts zero results. Manual smoke testing never catches this, because human testers ask reasonable questions and the failure only appears when someone asks an unreasonable one.

Silo, pool or bridge

AWS frames multi-tenant isolation as three patterns, and the trade-off is the same one you always get between blast radius and cost:

• Silo gives each institution its own knowledge base and its own vector index. Strongest isolation, easiest compliance conversation, and you multiply your idle infrastructure cost by the number of tenants. There is also a per-account service quota on knowledge bases, so check Service Quotas before you promise a hundred districts their own.
• Pool puts everyone in one index and relies entirely on metadata filtering. Cheapest by a wide margin, and one filter bug is a cross-institution disclosure rather than an embarrassment.
• Bridge splits the difference: separate knowledge bases sharing underlying vector infrastructure, which keeps the isolation story cleaner without paying for a full stack per tenant.

For a single university running many courses, pool with rigorous filtering is usually right, because the institution is the tenant and the courses are not. For an edtech vendor selling to multiple districts, I would start at bridge and only pool if the economics force it, because the conversation with a district’s data protection officer goes very differently when your answer is “we filter” rather than “they are separate indexes.”

Failure family three: guardrails that do less than you assume

Bedrock Guardrails is a useful layer and the first thing I reach for on an education workload: content filters, denied topics, word filters, PII detection and contextual grounding checks, applied at inference time and usable across models. For a system that teenagers will deliberately try to provoke, enforcement inside the invocation path beats anything bolted on afterwards. Three limits are documented and routinely missed.

Contextual grounding does not cover chatbots. AWS documents the supported use cases as summarization, paraphrasing and question answering, and states plainly that conversational QA and chatbot use cases are not supported. A tutoring assistant is a multi-turn conversation. If your hallucination story is “we turned on grounding checks,” read that page before it reaches a procurement document. The check still earns its place on single-shot retrieve-and-answer paths, which is where I would apply it, but a multi-turn tutor needs citation display and human review too.

PII detection is probabilistic and text-only. The sensitive information filter is a machine learning model that uses surrounding context to decide whether something is an identifier. It works better on prose than on bare fragments. AWS also documents that it operates on text output and does not inspect tool use parameters, so if your assistant calls a function with a student identifier in the arguments, that path is not covered. Do not treat it as a substitute for not putting identifiers in the prompt in the first place.

Blocked content lands in your logs. AWS documents that content blocked by guardrail policies appears as plain text in Bedrock model invocation logs when those logs are enabled. That is defensible for debugging and awkward when the blocked content is a distressed message from a fourteen year old. Decide deliberately whether invocation logging is on, where it lands, who can read it, and how long it lives. In education this is a governance decision, not a logging default.

None of this argues against Guardrails. It argues for knowing its edges, because the gap between what a feature name suggests and what the documentation says is where compliance commitments go to die.

Failure family four: the cost shape of an academic term

Education traffic is not a smooth curve. It is near zero over summer, spikes brutally in the week before every assessment deadline, and dies again. That shape punishes the default architecture in a specific way.

The console nudges you toward Amazon OpenSearch Serverless as the vector store. It bills per OpenSearch Compute Unit hour with a minimum baseline footprint that exists whether or not anyone queries it, so an empty index over a quiet August still bills. AWS has adjusted those minimums over time, so pull current figures from the OpenSearch Serverless capacity documentation rather than from any blog post, this one included. The mechanism is the part that matters: your floor is set by capacity that exists, not by traffic that happens.

Bedrock Knowledge Bases supports other backends, and the question that decides between them is idle cost:

• Aurora PostgreSQL with pgvector. A first-class documented option. Aurora Serverless v2 bills per Aurora Capacity Unit hour and can scale down to zero when idle, which fits the academic calendar almost too neatly. You take on index lifecycle work in exchange: choosing between HNSW and IVFFlat, keeping statistics fresh, watching bloat. If your institution already runs Postgres, that is work your DBAs recognize.
• Amazon S3 Vectors. Bills on storage and requests rather than provisioned capacity, so genuine idle costs approach nothing. The trade is retrieval latency, which is higher than a warm search cluster. For a tutor where a response already takes a second or two of model time, that may not be the bottleneck you think it is.
• OpenSearch Serverless. The right answer when retrieval latency is genuinely your constraint, when you need hybrid search and k-NN tuning, or when you already run collections for other workloads and the baseline is already paid for.

On the model side, the cost driver is retrieved context, not the question. Every turn ships the retrieved chunks as input tokens, so chunk size and the number of results retrieved multiply directly into your bill on every single message. Halving retrieved results is usually a bigger lever than switching to a cheaper model, and it often improves answer quality too, because the model has less irrelevant material to get distracted by.

Tag by course and faculty from day one. The first budget question you will be asked is which department is driving the spend, and retrofitting cost allocation tags after a pilot goes institution-wide is miserable. Cost Explorer answers it if the tags exist; tools like Vantage or CloudZero make the per-course view easier to hand to a finance team that does not want an AWS login.

Failure family five: no evidence when someone asks

FERPA governs education records in the United States, and the mechanism that lets a vendor touch them at all is the school official exception: the institution must maintain direct control over the vendor’s use and maintenance of the records. Similar logic applies under other regimes, and many districts and state contracts add data residency requirements that FERPA itself does not impose.

What that means architecturally is less exotic than it sounds, and it is mostly about being able to answer questions after the fact:

• Log the retrieval, not just the answer. Store which chunk identifiers were returned for each turn. When an instructor claims the assistant leaked a rubric, the retrieval log is the only thing that settles it.
• Keep student identifiers out of prompts. Pass an opaque session-scoped pseudonym and resolve it in your own system. The model does not need a real name to be helpful.
• Encrypt with customer managed KMS keys and use encryption context that names the tenant, so a key policy can actually constrain who decrypts what.
• Pin the region and enforce it with service control policies if residency is contractually required. An SCP is checkable evidence; a runbook is not.
• Write down retention for conversation history and enforce it with lifecycle rules. “We keep it forever because storage is cheap” is a defensible engineering position and an indefensible one in an education procurement review.
• Pull the current compliance reports and agreements from AWS Artifact rather than quoting a blog about what AWS certifies.

None of this is glamorous and all of it is cheaper to build in during the pilot than to retrofit when the institution’s legal team gets involved.

Troubleshooting the things that actually page you

“The new material isn’t showing up”

Almost always ingestion, not retrieval. Walk it in order rather than guessing. Did a sync actually run after the change? For an S3 data source, an edit is invisible until an ingestion job processes it.

aws bedrock-agent list-ingestion-jobs 
  --knowledge-base-id KB_ID 
  --data-source-id DS_ID

Look for a completed job starting after the upload. If the most recent job predates the change, your scheduler or event handler is the bug and retrieval is innocent. If a job did run, read its statistics: a modified-documents-indexed count of zero when you expected a change means change detection did not fire, which happens when an object is re-uploaded with identical bytes. A non-zero failed count means your file is probably the one that failed, and the failure reasons say why.

“The filter returns nothing at all”

Query once with no filter to confirm the content is indexed. If it comes back unfiltered and vanishes when filtered, it is a metadata problem: the sidecar was not ingested, the attribute name does not match exactly, or the value differs in case. These comparisons are literal. BIO201 and bio201 are two different courses as far as the filter is concerned, which is an argument for normalizing identifiers at both ingestion and query time in one shared helper.

“The launch shows a blank iframe”

Two suspects dominate. Cookies: your tool is in a cross-site iframe, so session cookies need the right SameSite and Secure attributes or the browser drops them and every launch looks like a fresh unauthenticated user. Registration mismatch: issuer, client ID or deployment ID differs between what the platform sends and what you stored. Log the received issuer and deployment ID on validation failure. Guessing at OIDC failures is slow; the values are right there.

“Answers are right but the citations are wrong”

Usually chunking. Slides and worksheets fragment badly under naive fixed-size chunking, and a chunk that starts mid-table cites a heading that belongs to something else. Before you change models, look at what the retriever actually returned for the query that misbehaved. Nine times out of ten the model was faithful to a bad chunk.

“Grades aren’t reaching the gradebook”

Check that a line item exists and that your service token carries the right AGS scopes. Passback failures are asynchronous, so they surface as an empty gradebook column weeks later rather than as an error the student sees. Alert on passback failure rate explicitly; nobody will report it in time.

Common mistakes

• Enforcing access control in the system prompt. Instructions are not permissions. If a chunk can be retrieved, assume it can be surfaced.
• Trusting client-supplied course or role values. Derive scope from the validated launch token and store it server-side, always.
• Ingesting the whole course folder. Instructor material and student material in one prefix is how rubrics leak. Separate at ingestion, tag at ingestion.
• Putting real names and student numbers in prompts. Pseudonymize before the model call and resolve afterwards.
• Sizing the vector store from peak week. The bill is set by the idle floor, and in education idle is most of the year.
• Assuming a guardrail feature does what its name implies. Read the supported use cases. Contextual grounding on a chatbot is the classic example.
• Piloting with one enthusiastic lecturer. That lecturer’s material is unusually well organized. The second department will not be.

Best practices

• Build the retrieval filter in exactly one function, and make the tenant and visibility clauses impossible to omit.
• Keep a small regression set of adversarial queries, including students asking for solutions and asking about classmates, and run it on every deployment.
• Show citations in the interface. Instructors trust the system faster when they can see the source, and it converts silent hallucinations into reportable ones.
• Give instructors a preview mode that answers as a student would see it. They will find scoping errors far faster than any test suite.
• Route the tool endpoint through a CDN and WAF such as Cloudflare, with rate limiting per session. Assessment week is a load test you did not schedule.
• Dashboard retrieval count, token spend per course and passback failure rate together. CloudWatch metrics fronted by Amazon Managed Grafana or Grafana Cloud gives faculty a view without handing out console access.
• Have an escalation path for distressing content that reaches a human being, and make sure the people on it know they are on it. This is a policy question you must answer before launch, not after.

Frequently asked questions

Do we need to fine-tune a model for our curriculum?

Almost certainly not. Curriculum content changes every term, and retrieval updates in minutes where fine-tuning is a rebuild. Fine-tuning earns its place for style and format consistency, not for facts. Start with retrieval, and revisit only if you have a specific behavior that prompting genuinely cannot produce.

How do we stop it from just doing the homework?

Partly retrieval scope, partly prompting, partly accepting you cannot fully win. Exclude solutions and rubrics from student-visible retrieval, prompt for Socratic responses on assessed material, flag the assessment context on the way in so the assistant knows which mode it is in. A determined student can paste the question into a consumer chatbot regardless, which is a course design question rather than an infrastructure one.

Is Bedrock the right choice versus calling a model API directly?

If the institution is already on AWS, Bedrock keeps the traffic inside your account boundary, gives you IAM and CloudTrail on the same footing as everything else, and can be reached over VPC endpoints. A direct provider API is often simpler and sometimes gets new models sooner. The deciding factor is usually procurement: one existing AWS agreement is a much shorter conversation than a new vendor review.

Does student data get used to train the model?

AWS states that Bedrock does not use customer inputs or outputs to train foundation models. Verify it against current AWS documentation and your agreement rather than any third-party summary, because this is exactly the clause a district will ask you to evidence in writing.

What does it cost to run a personalized learning assistant on AWS?

Model your bill as three parts: a vector store floor that bills whether or not anyone asks anything, a one-off embedding cost when you ingest the corpus, and per-turn inference dominated by retrieved context rather than by the question. For a small corpus with a term-shaped traffic curve, the floor usually dominates, which is why the vector store choice matters more than the model choice.

How do we know whether it actually helps anyone learn?

Decide the measure before launch and instrument for it. Usage is not learning. Pair the assistant’s logs with assessment outcomes on comparable cohorts and accept that a clean answer needs a study design, not a dashboard. Engineering can supply the data; it cannot supply the conclusion.

The one thing worth remembering

A personalized learning assistant on AWS will never tell you when it is wrong about who someone is. The model has no concept of enrollment, and retrieval has no concept of a due date. Both live entirely in the metadata you attach at ingestion and the filter you build server-side from a validated launch token. Get that right and what remains is ordinary engineering: chunking, cost curves, dashboards. Get it wrong and you ship something that works flawlessly right up until the week it embarrasses the institution.

Build the boundary first. Make it a control, not an instruction. Then go be clever with the tutoring.

Need a hand building or auditing one of these?

I work on the infrastructure side of AI assistants for education, which mostly means the parts that decide whether the thing is safe to put in front of students:

• Designing and reviewing the LTI 1.3 launch path, roster access and grade passback, including the cross-site cookie and registration issues that cause blank-iframe launches.
• Building the retrieval isolation layer: sidecar metadata schemas, server-side filter composition, and an adversarial regression suite that proves instructor material stays invisible to students.
• Choosing and migrating the vector store when the OpenSearch Serverless floor is eating a pilot budget, including Aurora pgvector and S3 Vectors options.
• Guardrail configuration that matches the documented behavior rather than the feature name, plus a logging and retention design you can defend in a procurement review.
• Cost allocation tagging by course and faculty, with dashboards that answer “which department is spending this” without handing out AWS console access.
• Infrastructure as code for the whole stack so a second institution is a variable file rather than a fortnight.

If you have something concrete, send it over: a retrieval filter you are not sure is airtight, an ingestion job that keeps skipping documents, a failing LTI launch log, or a Cost Explorer breakdown that stopped making sense. Easier to look at the real thing than to talk in the abstract.

The post Building a Personalized Learning Assistant on AWS Without Leaking Student Data appeared first on John Nessime.

That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.

The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.

What this post covers

Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn’t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.

Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.

Failure family one: there is no join key, only a filename

Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.

Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.

What to use instead

Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.

Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for “this exact bytes-on-disk artefact.” If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.

S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.

Failure family two: extraction confidence gets flattened into a fact

Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.

Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn’t go away. You just stopped being able to see it.

Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always “which extractor produced these,” and without that column the answer is a week of archaeology.

-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);

The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.

Failure family three: two catalogs, two permission models

This is the one that bites teams who did everything else right.

Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.

The result is predictable and unpleasant. An analyst who is correctly denied the salary column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.

There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:

• Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.
• Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.
• Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.
• Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.

Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with .metadata.json appended. That file is where classification, tenant, document ID and effective dates belong.

{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}

Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.

Failure family four: index decisions you cannot take back

S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.

By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.

If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.

# Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'

The dimension has to match your embedding model’s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.

Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.

Failure family five: reprocessing rewrites history

You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter’s board pack change.

This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.

Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and “why did this number move” has an answer you can query instead of guess at.

Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.

A shape that works for combining structured and unstructured data on AWS

Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.

1. Land raw, enable versioning, enable S3 Metadata. Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.
2. Mint the document ID at landing. Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.
3. Classify before you extract. Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.
4. Extract to a staging table, not straight to the curated table. Staging gives you somewhere to run validation before anything touches a table an analyst can query.
5. Merge into Iceberg on document identity, carrying confidence and provenance columns. Never merge on object key.
6. Embed separately, keyed by the same document ID. The vector plane and the table plane are different systems with different refresh rates. Treat them as such.
7. Join at query time in Athena. Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.

Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.

Troubleshooting

Athena says the principal has no privilege on the resource

This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.

Queries against metadata tables fail with a syntax error

Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.

Knowledge base sync fails partway with validation errors

Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.

Retrieval returns documents the caller should not see

Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.

Athena gets slower week over week on a table that isn’t growing much

Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.

Common mistakes

• Joining on the S3 object key because it was there and it worked in the prototype.
• Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.
• Applying a confidence threshold and discarding the score.
• Creating the vector index before deciding on the embedding model.
• Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.
• Reprocessing in place, with no record of which extractor produced which value.
• Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.

Best practices

• One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.
• Confidence and provenance columns are not optional. They are the difference between debugging and guessing.
• Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.
• Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.
• Enforce classification in all three planes independently. There is no single choke point.
• Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.
• Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.

FAQ

Do I need a lakehouse to combine structured and unstructured data on AWS?

No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.

Should extraction output live in the same table as the structured source data?

Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.

Is S3 Vectors a replacement for OpenSearch?

For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.

How do I keep the vector index in sync with the documents?

Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because “mostly in sync” degrades quietly.

What breaks first when this architecture is done badly?

Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.

Can I use Textract instead of Bedrock Data Automation?

Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.

Does Lake Formation govern my documents as well as my tables?

It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.

The one thing worth remembering

Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.

Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.

Need help wiring documents and tables together?

This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:

• Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.
• Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.
• Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.
• Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.
• Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.
• Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.

Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.

The post The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS appeared first on John Nessime.

Nothing. Not a cent. Oracle counts a vCPU on Amazon RDS exactly the way it counts a vCPU on Amazon EC2, and it says so in the same paragraph of the same policy document. The managed service does not buy you a discount on the software.

That single fact reshapes the whole argument. Once the license line is identical on both sides of the page, Oracle on EC2 vs RDS stops being a cost comparison and becomes a much more interesting question: which set of constraints can your application actually live inside? This post walks through the real differences, the places each option quietly hurts, the middle option that is being retired, and a decision procedure you can run before anyone opens a pricing calculator.

Start with the thing that is not a difference

Oracle publishes a policy called “Licensing Oracle Software in the Cloud Computing Environment.” It names a short list of Authorized Cloud Environments, and both Amazon EC2 and Amazon RDS are on it, alongside Microsoft Azure and Google Cloud Platform. Inside an Authorized Cloud Environment the counting rules are these:

• Two vCPUs count as one Oracle Processor license when multi-threading of processor cores is enabled.
• One vCPU counts as one Processor license when multi-threading is not enabled.
• The Oracle Processor Core Factor Table does not apply. The 0.5 multiplier you rely on for x86 hardware in your own data center is simply gone.
• For Standard Edition products, instances with four or fewer vCPUs count as one socket. Above four, every four vCPUs rounded up to the nearest multiple of four equals one socket.
• Standard Edition 2 may only be licensed on instances up to eight AWS vCPUs. That is a hard ceiling, and it is the single most common reason an SE2 estate has to become an Enterprise Edition estate on migration.

None of those rules distinguish EC2 from RDS. The policy treats them as the same environment for counting purposes.

There is one genuine licensing difference, and it is narrow. RDS offers a License Included model where AWS holds the Oracle license and bundles it into the instance-hour rate. That model covers Standard Edition 2 only. Enterprise Edition on RDS is bring-your-own-license, full stop. On EC2 there is no License Included option at all, so every Oracle database you run there is BYOL by definition.

Worth reading the footer of that Oracle policy document, too. It states that the document is for educational purposes and may not be incorporated into any contract. Your Oracle Master Agreement and ordering documents are what actually bind you. The cloud policy is how Oracle says it will behave, not what it promised in writing, and it can change without notice. Pin a copy of the version in force when you deploy and file it with your architecture decision record.

Find out what you are actually using first

Before you compare anything, find out which licensable options and management packs your current database is touching. Oracle tracks this itself. The feature usage view records every separately licensed feature the database has detected in use, which is exactly what an auditor looks at.

-- Which separately licensed features has this database touched?
SELECT name,
       detected_usages,
       currently_used,
       first_usage_date,
       last_usage_date
FROM   dba_feature_usage_statistics
WHERE  detected_usages > 0
ORDER  BY name;

Oracle also publishes an options and packs usage script through My Oracle Support that goes further and correlates usage across several dictionary views. If you have a support contract, use that one rather than rolling your own. The output frequently surprises people. Partitioning, Advanced Compression, and the Diagnostics and Tuning Packs get switched on by a well-meaning DBA or by a default setting and then sit there generating an entitlement gap.

Amazon RDS for Oracle: where it wins and where it hurts

Where RDS wins

• The operational floor is done for you. Automated backups with point-in-time recovery, minor version patching in a maintenance window, Multi-AZ synchronous standby with automatic failover, storage autoscaling. None of that needs a person on a rota.
• License Included removes the Oracle relationship entirely. For an SE2 workload where you own no licenses, this is genuinely hard to beat. There is no entitlement to track, no support renewal to chase, and no Oracle audit surface for the database software.
• Read replicas are configured for you. RDS manages the Data Guard setup behind cross-region replicas, including the network path between primary and standby. If the replica runs open read-only, that is Active Data Guard and you need the license for it, but the plumbing is not your problem.
• Parameter groups make configuration reviewable. Database parameters live in a versioned object you can diff, rather than in a spfile someone edited at two in the morning.

Where RDS hurts

The master user on an RDS for Oracle instance gets the DBA role with pieces removed. AWS documents exactly which privileges are withheld to keep the managed experience intact:

• ALTER DATABASE
• ALTER SYSTEM
• CREATE ANY DIRECTORY and DROP ANY DIRECTORY
• GRANT ANY PRIVILEGE and GRANT ANY ROLE

You also cannot log in as SYS, SYSTEM, or any other Oracle-supplied administrative account. This is the wall most migrations hit, and they hit it late. A packaged vendor application ships an installer that expects to connect as sysdba, create a directory object pointing at a local path, and run ALTER SYSTEM a dozen times. On RDS that installer does not complete. There is no workaround, because the privilege is not withheld by policy, it is withheld by design.

Three more constraints that bite later rather than sooner:

• Block size is fixed at creation. RDS for Oracle instances are created with an 8 KB default block size and it cannot be changed afterwards. The SYSTEM and temporary tablespaces always use it. You can create additional tablespaces at other block sizes by allocating the matching cache parameter and using the BLOCKSIZE clause, but the core of the database is fixed.
• Single files cap at 16 TiB. That is the ext4 filesystem limit on the underlying host, so bigfile data files inherit it. Try to grow past it and you get an ORA-01237 with a “File too large” error underneath.
• No host access at all. No shell, no third-party backup agent, no monitoring agent installed on the database server, no custom scripts on the filesystem. If your compliance tooling or your APM vendor requires an on-host agent, that requirement decides the platform for you.

Oracle on EC2: where it wins and where it hurts

Where EC2 wins

• Full sysdba and full root. Everything the vendor installer wants is available. Directory objects, external tables reading from local paths, custom PL/SQL that shells out, agents, whatever the packaged application assumed about its host.
• The full replication toolbox. Data Guard, Active Data Guard, and GoldenGate all run natively on EC2, and you control the protection mode, the apply lag tolerance, and the failover behaviour rather than accepting a managed default.
• Instance shape is a license lever. This is the underrated one. Oracle bills you on vCPUs, and most Oracle databases are memory bound rather than thread bound. On EC2 you can pick a memory-dense family and hit your SGA target without buying vCPUs, and therefore licenses, that the workload never uses. RDS gives you a narrower catalogue to choose from.
• You choose when to patch. No maintenance window imposed from outside, no forced minor version upgrade, no surprise reboot on a night your business cannot take one.
• Storage is yours to shape. Separate EBS volumes for data, redo, and archive logs, with independent IOPS and throughput. RDS abstracts that away.

Where EC2 hurts

Every one of those wins is a job someone now has to do, forever. The honest cost of Oracle on EC2 is not the instance-hour, it is the rota.

• Backups you have to prove. RMAN to S3 is a solved pattern, but a backup that has never been restored is a hypothesis. Schedule a real restore into a throwaway instance on a cadence and treat a failed restore as an incident.
• A standby that drifts silently. Data Guard apply lag is the classic invisible failure. Everything looks green, the standby is mounted, redo is shipping, and apply stopped three days ago because of a single nologging operation or a filesystem that filled. You find out during the failover.
• Patching is now a project. Quarterly Oracle patches, OS patches, and the interaction between them. On a Data Guard pair that means a standby-first sequence and a rollback plan.
• Monitoring you have to build. Tablespace growth, apply lag, wait events, ASM disk group headroom, archive log destination fullness. Amazon CloudWatch gets you host metrics and nothing about the database. Teams typically land on a Prometheus exporter feeding Grafana, either self-hosted or through Grafana Cloud or Amazon Managed Grafana, plus alerting that someone actually answers.
• Cost attribution gets murky. Instance, EBS, snapshots, cross-AZ traffic, and the license amortisation all land in different places. A cost visibility tool such as Vantage or CloudZero earns its keep here, because the Oracle line is the one finance will ask about.

The middle option that is being retired

For a while the answer to “I need sysdba but I also want managed backups” was Amazon RDS Custom for Oracle. It gave you shell access to the underlying EC2 instance and sysdba on the database, while keeping automated backups and host recovery, policed by a mechanism called the support perimeter that flagged your instance as unsupported if a change broke the automation.

AWS has announced that support for RDS Custom for Oracle ends on 31 March 2027. After that date the console and the resources go away, including instances, snapshots, and custom engine versions. AWS’s own guidance is to move those workloads to self-managed Oracle on EC2.

This is one of the few dates worth putting in an article, because it removes an option from your decision tree. If you are choosing a platform today, RDS Custom for Oracle is not a candidate. If you are already on it, the migration is on your roadmap whether you have written it down or not. Note that this retirement does not touch RDS for Oracle or RDS Custom for SQL Server.

The cost levers that actually move the number

Once you accept that the platform choice does not change the license rate, these are the things that do change the bill.

1. Memory-to-vCPU ratio. Reaching a memory target by moving up within a general purpose family buys vCPUs you did not need, and each pair of them is another Processor license. Picking a memory-optimised family instead is often the largest single saving available.
2. Multi-AZ under BYOL doubles the count. AWS states this plainly in its own documentation: under BYOL you must have a license for both the primary and the standby DB instance in a Multi-AZ deployment. Enabling the checkbox on a BYOL instance without buying the second set of licenses puts you out of compliance the moment it comes up.
3. Options and management packs. These are priced per Processor on top of the database license. On RDS, option groups can put some of them one click away, which is convenient and expensive in equal measure.
4. Edition, forced by the vCPU ceiling. SE2 stops at eight vCPUs in an Authorized Cloud Environment. A workload that needs more compute has to move to Enterprise Edition, and that is a different order of magnitude, not a percentage increase.
5. Infrastructure commitments. Reserved Instances and Savings Plans reduce the AWS side of the bill on both platforms. They do nothing to the Oracle side.

Two quick checks worth running before you model anything. First, confirm what an instance is actually presenting, because the multi-threading state is what decides whether you divide by two:

# On an EC2 instance: threads per core tells you the multi-threading state.
# "Thread(s) per core: 2" means the 2 vCPU = 1 Processor rule applies.
lscpu | grep -E 'Thread(s) per core|Core(s) per socket|^CPU(s):'

Second, check what your existing RDS instances are declaring, since the license model and the Multi-AZ flag together determine your exposure:

# Every Oracle RDS instance with its class, license model and Multi-AZ state.
aws rds describe-db-instances 
  --query "DBInstances[?starts_with(Engine, 'oracle')].
[DBInstanceIdentifier,Engine,DBInstanceClass,LicenseModel,MultiAZ]" 
  --output table

A MultiAZ value of true next to a LicenseModel of bring-your-own-license is the line to look at twice.

How I would decide between Oracle on EC2 vs RDS

Run these in order. The first question that returns a hard “no” makes the decision for you, and you can stop.

1. Does anything need sysdba or host access? Vendor installers, on-host agents, directory objects pointing at local paths, external tables reading local files, custom OS-level jobs. If yes, it is EC2. Nothing else in this list matters.
2. Run the feature usage query. Establish which options and packs you are on the hook for today. That number travels with you regardless of platform, and it is usually larger than the team expects.
3. Check the physical constraints. Non-8 KB block size on the SYSTEM tablespace, or a single data file heading past 16 TiB, both rule RDS out.
4. Settle the edition. If SE2 fits inside eight vCPUs and you own no perpetual licenses, RDS with License Included is very hard to argue against. If you need Enterprise Edition, you are BYOL on either platform and the licensing argument is over.
5. Count licenses at the shape you would really run. Production instance, plus standby if you want Multi-AZ or Data Guard, plus every option pack from step two. Do this before you look at instance-hour rates, because it is the bigger number.
6. Price the rota honestly. Who runs restore tests, who owns the patch calendar, who gets paged at three in the morning when apply lag alerts. If the answer is “we will figure it out,” that is a vote for RDS.
7. Pick, then write down why. Record the Oracle policy version you counted against and the instance shapes you assumed. When an audit letter arrives, that document is the difference between a conversation and a negotiation.

One useful side effect of building a dev or reporting copy outside AWS entirely: a modest VPS from a provider like Contabo or InterServer running Oracle’s free Express Edition gives developers a schema to work against without touching a licensed instance. It will not stand in for production, but it takes pressure off the environments that cost money.

Arguments that do not survive contact

• “RDS is cheaper because AWS manages it.” Under BYOL the license line is identical. What you save is operational labour, which is a real saving but a different budget line, and you should say which one you mean.
• “We will disable multi-threading to reduce licenses.” Turning it off halves your usable threads and flips the ratio to one vCPU per Processor license. You end up at the same license count with less compute.
• “The core factor gives us 0.5 on x86.” Not in an Authorized Cloud Environment. Oracle states the table does not apply there. This is the source of the widely repeated claim that moving to the cloud doubles Oracle licensing on identical hardware.
• “The standby is passive, so it is free.” AWS documents that BYOL Multi-AZ requires licensing both instances. There is a separate Oracle provision about genuinely cold failover nodes, but a synchronous standby that is always running is not that. Get your specific position confirmed in writing by whoever advises you on Oracle contracts, not from a blog, including this one.
• “Graviton is a different architecture, so it licenses differently.” The cloud policy counts vCPUs. It does not carve out processor architectures on shared tenancy.
• “We will start on RDS and move to EC2 if it does not work.” Technically true and practically expensive. Moving off RDS means a full migration project with a cutover window, not a configuration change. Establish the sysdba requirement before you build, not after.

Frequently asked questions

Is Oracle licensing cheaper on RDS than on EC2?

Under bring-your-own-license, no. Oracle’s cloud policy names EC2 and RDS together and applies the same vCPU counting to both. The only structural difference is that RDS offers a License Included model, which is available for Standard Edition 2 only.

Can I run Oracle Enterprise Edition with License Included on RDS?

No. AWS supports License Included for Standard Edition 2 only. Enterprise Edition on RDS is BYOL, and you need the appropriate licenses with active Software Update License and Support.

Do I need to license the Multi-AZ standby?

Under BYOL, AWS documentation states you must have a license for both the primary and the standby DB instance. Budget for double the Processor count before you enable it. Under License Included the standby is reflected in the instance-hour rate instead.

What is the largest Standard Edition 2 instance I can run on AWS?

Oracle’s cloud policy caps Standard Edition 2 at eight AWS vCPUs. Past that you are into Enterprise Edition. This ceiling catches a lot of migrations that were sized on memory requirements rather than vCPU count.

Can I get sysdba access on RDS for Oracle?

No. The master user receives the DBA role minus a documented set of privileges including ALTER SYSTEM, ALTER DATABASE, CREATE ANY DIRECTORY, and the GRANT ANY privileges, and the Oracle-supplied administrative accounts are not available for login. If an application requires sysdba, EC2 is your platform.

Is RDS Custom for Oracle still a viable middle ground?

Not for new work. AWS ends support for RDS Custom for Oracle on 31 March 2027 and recommends moving those workloads to EC2. Existing users should treat the migration as a scheduled project rather than a future problem.

Does turning off hyperthreading reduce my Oracle license count?

No. With multi-threading enabled the ratio is two vCPUs per Processor license. With it disabled the ratio becomes one to one. The license requirement stays where it was while your available compute halves.

Conclusion

If you remember one thing from the Oracle on EC2 vs RDS comparison, make it this: the platform choice does not change what Oracle charges you. It changes who does the work and what your application is allowed to do.

So run the decision in that order. Establish the sysdba requirement first, because it is binary and it ends the discussion. Count your options and packs second, because that number is usually wrong in the direction that costs money. Compare instance-hour rates last, because it is the smallest term in the equation and the one everyone starts with.

And write down which version of Oracle’s cloud policy you counted against. It is a non-contractual document that can change, and the copy you filed on the day you deployed is worth more than any argument you can make from memory later.

Need a second pair of eyes on an Oracle migration to AWS?

Most of the expensive mistakes here happen before anything is built, in the assumptions nobody wrote down. Things I can help with:

• Auditing feature and option pack usage on your existing databases so you know your real entitlement position before you size anything
• Working through the sysdba and host-access requirements of packaged applications to determine whether RDS is even on the table
• Modelling Processor counts across candidate instance shapes, including standby and Multi-AZ exposure, so the number in the business case survives review
• Building the operational layer that self-managed Oracle on EC2 needs: RMAN to S3, tested restores, Data Guard apply-lag alerting, and patch runbooks
• Planning migrations off RDS Custom for Oracle to EC2 ahead of the end-of-support date
• Setting up monitoring and cost attribution that separates the AWS line from the Oracle line, so finance gets an answer instead of a total

Send me the output of your feature usage query, or a list of the instance classes you are considering, and I will tell you what I would check first.

The post Oracle on EC2 vs RDS: Why the License Math Doesn’t Save You appeared first on John Nessime.

That gap is the whole problem with permission cleanup. IAM Access Analyzer unused access findings are genuinely useful, and they are also built on evidence with known holes in it. AWS says so in its own documentation. Treat a finding as proof rather than as a lead and sooner or later you take something down.

This post covers what an unused access finding actually asserts, the four blind spots that turn a clean-looking finding into an outage, and a removal procedure that fails safe.

What an unused access finding actually claims

An unused access analyzer is a separate analyzer type from the external access one. You create it per account or across an organization, and it produces four kinds of finding: unused roles, unused IAM user access keys, unused IAM user passwords, and unused permissions at the service and action level for principals that are active.

aws accessanalyzer create-analyzer 
  --analyzer-name unused-access-prod 
  --type ACCOUNT_UNUSED_ACCESS 
  --configuration '{"unusedAccess": {"unusedAccessAge": 90}}'

unusedAccessAge is the threshold in days beyond which access counts as unused, and it accepts 1 to 365. Use ORGANIZATION_UNUSED_ACCESS from a delegated administrator account. Unused access findings don’t vary by Region, so you don’t need one analyzer per Region.

Here is the part that matters. The analyzer does not replay CloudTrail and reconstruct what your workload did. It reads IAM last accessed information, the same data behind the Access Advisor tab in the console. Everything the analyzer knows, and everything it doesn’t, comes from there. The blind spots below are last accessed blind spots, inherited wholesale.

The four blind spots that break production

1. Action-level tracking doesn’t cover everything

Action last accessed information only exists for the services and actions AWS has explicitly added to the tracked list, and it covers management actions. The IAM console spells out one case in plain text: action last accessed information is not available for Amazon S3 data events.

Think about what that means for a data pipeline. A role whose entire purpose is s3:GetObject and s3:PutObject against one bucket is doing object-level work all day. Bucket-level calls like ListBucket are management events and get tracked. The object-level calls that constitute the actual job are data events, and they sit outside action last accessed.

AWS puts the warning in the documentation itself: do not make permissions decisions based solely on the absence of tracking information. Almost nobody reads that sentence.

2. Denied attempts count as access

Service last accessed data includes all attempts to call an AWS API, not just successful ones. A role that has been hammering a service it has no permission for, failing every time, shows up as having used that service.

This runs opposite to the first blind spot, and it is why a findings list can be simultaneously too aggressive and too timid. Dead permissions look alive because a misconfigured client keeps retrying against them, so you clean up around them and leave the real cruft in place.

3. The calendar is longer than your window

A ninety-day tracking period is the common default and it is wrong for a predictable class of role:

• Disaster recovery and failover roles, exercised once or twice a year
• Quarterly close, audit, and regulatory reporting roles
• Annual key rotation or certificate renewal automation
• Break-glass roles, which should be unused by design
• Seasonal workloads that only wake up for a peak period

There is also a ceiling worth knowing. unusedAccessAge tops out at 365 days, while IAM reports last accessed activity for at least the trailing 400 days. The analyzer cannot be configured to span the full window the underlying data actually holds, so for anything on an annual cycle it will not settle the question on its own.

4. Service-level use says nothing about resource-level need

A finding operates on services and actions. Your policy probably operates on services, actions, resources, and conditions. Knowing a role used secretsmanager:GetSecretValue last week does not tell you which secret, and narrowing the policy to a specific resource ARN based on a finding is a guess dressed up as evidence.

This is where policy generation earns its place. It reads CloudTrail activity for a principal over a window and drafts a policy from what it observed, including resource-level detail where the log has it. Same caveat: policy generation does not identify action-level activity for data events either. Better first draft, not an oracle.

A removal procedure that fails safe

The core idea: deny before you delete. An explicit deny is one API call to add and one to remove, effective immediately in both directions. Deleting a policy statement or a role is not symmetrical; restoring it means someone reconstructing intent under pressure. Use deny as a reversible dry run for deletion.

1. Sort findings by blast radius, not by count. An unused permission on a role assumed by one Lambda is a different risk from one on a role fifty engineers assume daily.
2. Confirm the role’s activity independently before touching permissions.
3. Attach a scoped deny as an inline policy on the role, leaving the existing allow untouched. An explicit deny wins over any allow, so this simulates removal exactly with none of the loss.
4. Watch CloudTrail for denials attributed to that role. This is the observation window and it is the step people skip.
5. Wait through at least one full business cycle, longer if the workload has monthly or quarterly rhythm.
6. Only then edit the allow and remove the deny.

For step two, the cheapest independent check is the role’s own last-used timestamp:

aws iam get-role --role-name legacy-reporting-role 
  --query 'Role.RoleLastUsed'

That returns LastUsedDate and the Region where the role was last used. LastUsedDate is null if the role has not been used within the tracking period, which is not the same as never used. The Region field is quietly valuable: a supposedly dead role showing activity in a Region nobody works in is worth a conversation before a deletion.

For step three, the deny is deliberately narrow. Deny the flagged actions on the flagged service and nothing else:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CanaryDenyUnusedRedshift",
      "Effect": "Deny",
      "Action": "redshift:*",
      "Resource": "*"
    }
  ]
}

Give it a Sid you can grep for, because in six months somebody will find this policy and have no idea whether it is a canary or a control.

For step four, you want CloudTrail records where the request failed on authorization. The field is errorCode, and the values that matter are AccessDenied plus the EC2-specific Client.UnauthorizedOperation. Filter on the assumed-role session issuer so you catch every session. CloudTrail Lake or Athena over your trail bucket both work.

To make this an ongoing signal rather than a one-off query, wire a CloudWatch Logs metric filter on AccessDenied to a dashboard. Amazon Managed Grafana, self-hosted Grafana, or Grafana Cloud all render it fine. What matters is that it is broken down by principal, so a spike points at a role rather than at the account.

Deleting the role itself has a separate trap

When you name a principal by ARN in a policy, IAM stores the underlying unique principal ID rather than the string. Delete that role, recreate it with the identical name, and you get a new unique ID. Every policy that referenced the old one now points at something that no longer exists. Nothing errors at deletion time. The break surfaces later, in another account, when a cross-account assume-role call starts failing and the trust policy on screen looks exactly correct.

Before deleting a role, check the things a findings list will never tell you:

• Is it named as a principal in any resource policy, S3 bucket policy, KMS key policy, or another account’s trust policy?
• Is it attached to an instance profile?
• Is it managed by CloudFormation, Terraform, or OpenTofu? Deleting out of band leaves drift that surfaces on the next apply.
• Is it a service-linked role? Those are owned by a service and have their own deletion procedure.
• Is it the execution role for a Lambda function, ECS task, or Step Functions state machine that runs rarely?

Safe order: detach the permissions, leave the role and its trust policy in place, wait another cycle, then delete. An empty role is harmless. A missing one is not.

Keeping the findings list small enough to read

A findings list nobody triages is worse than none, because it produces the appearance of coverage. Two levers keep it honest. The first is exclusions, which can skip whole accounts or skip principals by tag, and can be updated at any time:

aws accessanalyzer update-analyzer 
  --analyzer-name unused-access-prod 
  --configuration '{
    "unusedAccess": {
      "analysisRule": {
        "exclusions": [
          {"accountIds": ["111122223333"]},
          {"resourceTags": [{"lifecycle": "break-glass"}]}
        ]
      }
    }
  }'

Tag-based exclusion is the one to reach for. Sandbox accounts churn roles and generate noise with no security value; break-glass roles are supposed to look unused. Excluding by tag survives new accounts joining the organization. An account ID list does not.

The second lever is archiving findings you have consciously accepted:

aws accessanalyzer update-findings 
  --analyzer-arn <your-analyzer-arn> 
  --ids <finding-id> 
  --status ARCHIVED

Archiving is a decision record, not a dismissal. Write down why somewhere durable, because the same finding will come back to a different person.

Scope has a cost dimension too. Unused access analysis is billed on the number of IAM roles and users analyzed per month, per analyzer, so a second analyzer with a different tracking period doubles the principals counted. If you watch spend through Cost Explorer or a layer like Vantage or CloudZero, confirm where this line lands before widening scope organization-wide.

Troubleshooting

No findings yet, or a finding that won’t clear after you fixed it. After creating or updating an analyzer it takes time for findings to appear, recent activity takes around four hours to reach last accessed data, and scans are periodic rather than instant. Give it a full cycle. If you use the CLI or API for last accessed reports, generate a fresh one; the old report will not update itself.

The finding says unused but the team insists otherwise. Assume the team is right and the tracking has a hole. Check whether the work is data-plane activity, then whether the actions involved are on the tracked list at all. Absence of tracking data and absence of use look identical from the findings page.

Your canary deny broke something unexpected. That is the canary working. Detach the inline policy, confirm the failures stop, then narrow the deny to specific actions rather than the whole service and run it again.

Something broke but no AccessDenied appears in CloudTrail. Check whether the failing call is a data event. A trail logs management events by default; data event logging is opt-in per resource type and carries its own charges. No configured data events, no record of the denial.

Common mistakes

• Treating a finding as evidence of non-use rather than absence of recorded use. Different claims.
• Deleting the role in the same change as removing its permissions, collapsing two reversible steps into one that isn’t.
• Setting the tracking period once and never revisiting it against the workload’s actual rhythm.
• Cleaning up in the console while Terraform or CloudFormation remains the source of truth, so the next apply puts it all back.
• Batching removals across many roles at once. When something breaks you have a bisect problem instead of a rollback.

Best practices for IAM Access Analyzer unused access cleanup

• Tag principals with their expected usage cadence at creation time. It costs nothing then and answers the hardest question later.
• Start with unused access keys and passwords on IAM users. Highest value, lowest risk removals in the whole list.
• Route findings to EventBridge and into your ticket queue so cleanup is tracked work rather than a periodic heroic sweep.
• Make the permissions change in code, apply it through the normal pipeline, and let the diff be the record.
• Keep an AccessDenied dashboard permanently, not just during a cleanup. Cheapest early warning you can build for this class of change.

Frequently asked questions

Does IAM Access Analyzer read my CloudTrail logs?

Not for unused access findings. Those come from IAM last accessed information. Policy generation is the feature that reads CloudTrail, and it is a separate operation run against a specific principal and time range.

What tracking period should I set?

Ninety days is reasonable for interactive human roles and continuously running workloads, and too short for anything on a quarterly or annual cycle. The value is configurable from 1 to 365 days. The right answer is a property of your workload, not a security best practice number.

Can a finding be wrong?

It can mislead in both directions. Permissions used only through data events can appear unused, and permissions that only ever produced denied attempts can appear used, because last accessed data counts all attempts rather than only successful ones.

Do I need an analyzer in every Region?

No. Unused access findings do not change based on Region, so one per account, or one at the organization level, is enough. External access analyzers are the ones with a Region dimension.

How is unused access analysis billed?

On the number of IAM roles and users analyzed per month, per analyzer. Scope is the cost lever, which is why exclusions matter for the bill and not just the noise. Check current published rates before enabling it organization-wide.

Should I delete unused roles or just strip their permissions?

Strip first, delete later, in two separate changes. A role with no permissions has almost no attack surface, and keeping it alive preserves any principal references pointing at it while you find out if anything still cares.

Conclusion

IAM Access Analyzer unused access findings answer a narrower question than the one you care about. They tell you what AWS did not record activity for. They do not tell you what your workload does not need, and they cannot tell you whether a permission that is used should be. The distance between those statements is where the outages live.

The one thing worth remembering: deny before you delete. An explicit deny gives you the exact behaviour of a removal with a one-call undo, and that single habit turns permission cleanup from a change you have to be brave about into one you can be systematic about.

Need help trimming IAM permissions without the outage

The risk in permission cleanup sits almost entirely in what the tooling cannot see. Things I can help with:

• Setting up account or organization-level unused access analyzers with tracking periods and tag-based exclusions that fit how your workloads run
• Building the AccessDenied observability layer, from CloudTrail queries through metric filters to a dashboard the team will actually look at
• Running staged deny-then-delete cleanups with the rollback path defined before anything changes
• Auditing roles for cross-account principal references, instance profiles, and IaC ownership before deletion
• Moving console-era IAM into Terraform or OpenTofu so the next cleanup is a reviewable diff

If you have a findings list you are not sure how to act on, send me the role’s policy JSON and the finding detail and I will tell you what I would check before touching it.

The post IAM Access Analyzer Unused Access: Removing Permissions Without Breaking Production appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

The part that hurts you later is not on that page.

The AWS Schema Conversion Tool for Oracle grades syntax. It tells you whether it could emit valid PostgreSQL for an object. It does not tell you whether that object will behave the same way at runtime. A trigger can convert with a green tick and fire in a different order. A column can convert with a green tick and silently drop the time component off every date in the table. A procedure can convert with a green tick and run twenty times slower because every string function call now goes through a wrapper.

This post sorts the tool’s output into four buckets: what converts cleanly, what converts while quietly changing meaning, what converts only through the extension pack, and what it refuses outright. Then the category nobody budgets for, which is everything it never looks at.

What the AWS Schema Conversion Tool for Oracle actually does

It is a Java desktop application. It connects over JDBC to source and target, reads the Oracle data dictionary, parses the DDL and PL/SQL it finds, and generates equivalent PostgreSQL objects and PL/pgSQL. The converted code stays inside the project until you explicitly apply it to the target or save it out as SQL files.

Three things follow from that description, and they matter more than any feature list.

• It only sees what is stored in the database. Code that lives in your application, your job scheduler, or a reporting tool is invisible to it unless you point the separate application SQL conversion feature at those files yourself.
• It only sees what the connecting user can see. Connect with a thin account and half the schema will not appear in the report at all.
• It moves structure and code. It does not move rows. That is AWS DMS, and it is a separate exercise with its own failure modes.

There is also a command line interface, which is the part most teams discover too late. It uses scenario files rather than flag-style arguments, and it needs Amazon Corretto 11 installed alongside the tool.

# Interactive mode, run from the AWS SCT app folder.
# End each command with a / on its own line.
java -jar AWSSchemaConversionToolBatch.jar -type interactive

# Script mode, running a saved .scts scenario end to end.
java -jar /opt/aws-schema-conversion-tool/lib/app/AWSSchemaConversionToolBatch.jar 
  -type scts -script /home/sct/project/convert.scts

Script mode turns a one-off conversion into something you can rerun, which matters because on a real migration you will convert the schema many times as you clean up the source. Re-clicking through the GUI each time is how mistakes creep in.

Tier one: converts cleanly, and you can trust it

Credit where it is due. For the bulk of an ordinary OLTP schema the tool is good, and hand-writing this would be miserable.

• Table definitions: names, columns, ordinary data types, NOT NULL, defaults, primary keys, unique constraints, foreign keys.
• Standard b-tree indexes, including most single-column functional indexes.
• Ordinary views. The basic CREATE VIEW syntax is close to identical between the two engines.
• Sequences and identity columns, despite the syntax looking nothing alike.
• ANSI-standard SQL, joins, subqueries, and almost all window and analytic functions.
• PL/SQL flow control: loops, blocks, conditionals, exception structure.
• The four standard transaction isolation levels, plus BEGIN, COMMIT, and ROLLBACK.

If your schema is mostly tables, views, and reporting SQL with modest procedural code, the automation figure in the report is roughly honest and you should believe it.

Tier two: converts green, behaves differently

This is the tier that costs money, because none of it shows up as a red action item. The object converted. The DDL applied. The application connected. Something is wrong three weeks later and nobody thinks to look back at the schema conversion.

Oracle DATE is not PostgreSQL date

Oracle’s DATE type carries a time component down to the second. PostgreSQL’s date is a calendar day and nothing more. Map an Oracle DATE to a PostgreSQL date and every timestamp in that column is truncated, permanently, without an error. The correct target is timestamp. Check the conversion settings and check the generated DDL before you apply, because this one is not recoverable after the data lands.

NUMBER without precision becomes numeric, and numeric is slow

An unqualified Oracle NUMBER is an extremely flexible container, so the default mapping to PostgreSQL numeric is technically correct. It is also arbitrary-precision arithmetic implemented in software. Put that on a surrogate key that participates in every join and index in your application and you have bought yourself a performance problem that no amount of index tuning will fix.

Find the candidates before conversion, not after:

-- Oracle: unqualified NUMBER columns, the ones that default to numeric.
SELECT table_name, column_name, data_precision, data_scale
FROM   user_tab_columns
WHERE  data_type = 'NUMBER'
AND    data_precision IS NULL
ORDER  BY table_name, column_name;

For each one, look at the actual minimum, maximum, and scale of the data, then decide whether it is really an integer, a bigint, or a genuine numeric(p,s). Surrogate keys and foreign keys are almost always integers pretending to be NUMBER. Money columns usually are not.

Empty string is NULL in Oracle and is not in PostgreSQL

This is the single most reliable source of post-cutover bugs, and it is not a conversion defect. Oracle treats an empty string as NULL. PostgreSQL treats it as a zero-length string, which is a real value. Every IS NULL check in your procedural code and your application was written under Oracle’s rule.

Concatenation flips too. In Oracle, concatenating a NULL is a no-op. In PostgreSQL, concatenating a NULL with the || operator yields NULL for the whole expression. A report that used to print a name with a missing middle initial now prints nothing at all.

After the data lands, count the divergence before you let users near it:

-- PostgreSQL: how many rows are empty string rather than NULL?
SELECT count(*) FILTER (WHERE middle_name = '')   AS empty_string,
       count(*) FILTER (WHERE middle_name IS NULL) AS real_null
FROM   app.customer;

If empty_string is non-zero anywhere, you have a decision to make: normalize the data on load, or audit every NULL check that touches those columns. Normalizing is usually cheaper and is far easier to verify.

The warnings the tool does raise, that people scroll past

Some of these semantic differences do appear in the action list, just not in red. They are worth reading individually rather than in aggregate.

• GREATEST and LEAST are flagged as potentially producing different results from the source, because of how each engine handles NULL arguments and type coercion.
• Timestamp handling inside converted partitioned tables is flagged as potentially divergent.
• Converted code is flagged as possibly incorrect where bind variable names, user-defined functions, or dynamic SQL are involved. That is the tool telling you it guessed.
• Trigger firing order is not carried over. Oracle’s FOLLOWS and PRECEDES clauses have no equivalent, and PostgreSQL fires triggers on the same event in alphabetical order by trigger name. If your Oracle triggers depended on sequencing, you now depend on naming.
• An empty exception block in the converted output is called out explicitly. Oracle code that swallowed errors will now swallow them differently.

One more that is not an action code at all: Oracle PL/SQL routinely issues COMMIT in the middle of a procedure. A PostgreSQL function cannot do that. Either the logic becomes a procedure invoked with CALL, or the transaction boundary moves out into the calling application. That is a design decision, not a find-and-replace.

Tier three: converts, but only through the extension pack

When there is no native PostgreSQL equivalent for something Oracle provides, the tool installs an extension pack: an additional schema on the target, named aws_oracle_ext for Oracle sources, containing wrapper functions, tables, and supporting objects that emulate the source behavior. For some capabilities like job scheduling and sending mail, the pack also deploys AWS Lambda functions.

So a call to INSTR in your PL/SQL comes out the other side as aws_oracle_ext.INSTR. Package variables, which have no PostgreSQL equivalent because PL/pgSQL has no packages, get routed through getter and setter functions in the pack.

The case for accepting this is real. If the goal is off Oracle licensing by a fixed date, the pack converts code that would otherwise need a developer per object. That is the difference between a migration that ships and one that stalls in refactoring, and nobody gets a bonus for elegant PL/pgSQL on a project that missed its window.

The case against is equally real:

• Wrapper functions sit between your code and the engine. In hot loops, that overhead is measurable, and the package-variable emulation in particular is not free.
• You have swapped one dependency for another. The pack is AWS-specific code living in your database, which is an awkward thing to explain if the stated goal was ending vendor lock-in.
• Every time the pack is applied, its components are overwritten. Anything you hand-tuned inside that schema is gone.
• Upgrades and target-version changes now have an extra moving part to validate.

There is a middle path. The tool can be configured to use the open-source orafce extension, which implements a subset of Oracle functions and packages natively in PostgreSQL, in place of some pack wrappers. My rule of thumb: use the pack to get the migration over the line, then treat the size of the aws_oracle_ext call graph as a debt register you burn down deliberately, starting with whatever appears in your slowest queries. Rewriting collections as native arrays or JSON, and package variables as parameters, buys back most of the performance.

Tier four: it stops and hands you the work

These are the red items: the honest part of the report, and paradoxically the least dangerous, because you can see them and plan around them. Not exhaustive, but this is what turns up on real Oracle schemas.

• Database links. No automation at all. The mechanism has to be rebuilt, usually on postgres_fdw, and foreign data wrappers do not support calling user-defined functions on the remote side. If your architecture leans on links, this is a workstream, not a task.
• Autonomous transactions. PostgreSQL has no direct equivalent. Logging patterns that relied on committing independently of the parent transaction need redesign, often with dblink back to the same instance or an out-of-band queue.
• ROWID and UROWID. No equivalent, and any code that reads ROWID from a table is flagged separately. Row-identity tricks in PL/SQL are a rewrite.
• Dynamic SQL and DBMS_SQL. Anything assembled at runtime the parser cannot resolve statically comes back untouched or with a health warning.
• Synonyms. Not supported, and neither is their usage, so every reference through a synonym has to be resolved by hand or mimicked with views and search paths.
• Global temporary tables, index-organized tables, external tables, clustered tables, object tables, nested tables, virtual columns, BFILE. Storage and type features with no target equivalent.
• Bitmap indexes, domain indexes, multi-column functional indexes. The first two have no counterpart; the third is a documented conversion gap.
• Associative arrays, VARRAY of VARRAY, multi-dimensional arrays, object type methods, member functions in user-defined types. Collection-heavy PL/SQL is where the automation percentage collapses.
• Compound triggers, system triggers, REFERENCING clauses, conditional predicates. The trigger model differs enough that anything beyond a plain row trigger needs attention.
• Java stored routines and table functions. Not converted.
• Query hints. Partially converted, and you should delete them all first anyway. Hints tuned for Oracle’s optimizer are noise to a different planner. Add them back only after schema, index, and query work have failed.
• Materialized view incremental refresh and DML on materialized views. The view itself may convert; the refresh strategy will not.

Some red items are more conservative than your actual target

Here is a detail that saves real budget. The action code index and the migration playbook were written against a baseline target, and PostgreSQL has moved since. MERGE is listed as unsupported, but PostgreSQL has had a native MERGE statement since version 15. FOR UPDATE SKIP LOCKED is listed as unsupported, and it has been in PostgreSQL for a long time. GROUPING SETS, CUBE, and ROLLUP are listed as unsupported and are also long-standing PostgreSQL features.

What this means practically: an action code tells you the converter will not emit the construct for you. It is not a statement about your target engine. Before you scope a rewrite for anything on that list, check the feature against the PostgreSQL version you are actually deploying. Some of those “rewrites” are a copy-paste with a syntax tweak.

What it never looks at

This is the category that wrecks timelines, because it is not in the report at all. Not green, not red. Out of scope, therefore invisible.

• Application code. There is a separate application SQL conversion feature that scans source files for embedded SQL, but it is opt-in, file-based, and you have to aim it. Nothing happens automatically.
• Scheduled work. DBMS_SCHEDULER jobs, cron entries calling SQL*Plus, shell wrappers, ETL controllers. The schema converts; the thing that runs it every night does not.
• Grants, roles, profiles, and proxy authentication. Your security model is a separate design exercise on the target.
• Oracle features with no schema footprint. Data Guard, RAC, ASM, RMAN, Oracle Text indexes, fine-grained access control policies. Each is its own replacement decision.
• Optimizer statistics and execution plans. Nothing about performance carries over. You are starting from zero on the target, and that is the right expectation to set with stakeholders early.
• Everything downstream. BI dashboards, reporting layers, and integration jobs that embed Oracle-flavored SQL will break on the day you cut over, and none of them appear in the assessment.

Inventory these before conversion, not after. A one-page list of every consumer of the schema is worth more than a week of tuning.

Desktop tool or DMS Schema Conversion?

DMS Schema Conversion is the managed, browser-based version built on the same engine. Instead of a laptop install you create an instance profile, register source and target data providers with credentials in Secrets Manager, and run the project from the console, with metadata staged in S3.

AWS now points standard OLTP conversions at the managed option, and generative AI assisted conversion is available there on supported paths. The desktop tool remains available and still covers ground the managed feature does not.

• Pick the desktop tool when you need scripting through the CLI, want to edit the SQL for source and target inside the project, need migration rules and tree filters, are converting a data warehouse target, or want the application SQL and ETL conversion features.
• Pick DMS Schema Conversion when your source is reachable from a VPC, you would rather not manage a workstation with database credentials on it, your conversion path and region are supported, and the generative AI assist is useful to you.

Check supported paths, versions, and regions before committing either way; they differ between the two. On the desktop side, note the supported operating systems are Windows, Fedora, and Ubuntu, all 64-bit. If your analysts are on Macs, plan for a Linux jump box. A small VPS from a provider like Contabo or InterServer, or an EC2 instance inside the VPC, is the usual answer, and it has the side benefit of keeping the JDBC connection close to the source rather than dragging a dictionary read across a home broadband link.

If the Oracle source is on-premises and the conversion box is not, you need a network path that a DBA will sign off on. That is normally a site-to-site VPN or Direct Connect. For an individual consultant working against a client’s staging environment, a commercial VPN such as NordVPN or Surfshark is sometimes what is on offer, but treat it as a stopgap: it gets you connectivity, not an audit trail.

Troubleshooting

The CLI jar refuses to open

A common failure is an unexpected error opening AWSSchemaConversionToolBatch.jar. Recent OpenJDK builds cap the size of jar signature files and the SCT jar exceeds the default. Raising that limit on the java invocation is the widely reported workaround:

java -Djdk.jar.maxSignatureFileSize=20000000 
  -jar AWSSchemaConversionToolBatch.jar -type interactive

Confirm you are on the Corretto 11 runtime the tool expects before chasing anything more exotic.

It hangs on a large schema, or will not connect at all

Thousands of objects in one schema puts the JVM under memory pressure; Global settings exposes JVM options, so raise the heap and retry on a machine with real memory. Separately, JDBC driver paths live in Global settings under Drivers, per engine. A driver sitting on disk but not registered there fails in a way that looks like a network problem and is not.

“The package body doesn’t include source code”

The source is wrapped. Obfuscated PL/SQL cannot be parsed, so it cannot be converted. Either find the original in version control or accept a from-scratch rewrite. Finding this on a critical package late in the project is a bad day, so check for wrapped objects on day one.

Objects will not apply, or the report is missing things

Failures on apply are usually dependency order: an object references a parent that was not converted or sits outside the schema mapping, and the action list says so explicitly. Convert the whole dependency closure, not just the objects you care about. A short report is usually privileges, since the tool reads through the connecting account’s visibility. Compare its object counts against a direct dictionary count before trusting any percentage in it.

Common mistakes

• Treating the automation percentage as a completion estimate. It measures objects, not effort, and one gnarly package can outweigh four hundred clean tables.
• Applying to the target before reviewing the generated DDL. Save to SQL, read it, diff it on the next run.
• Accepting default type mappings wholesale. DATE and unqualified NUMBER are the two that reliably hurt.
• Carrying Oracle hints across. Start with none and earn each one back.
• Scoping rewrites straight off the action codes without checking your target PostgreSQL version first.
• Leaving project files and scenario scripts containing database credentials sitting on a laptop after the engagement. They are plain files; when the work is done, wipe them properly with a tool like O&O SafeErase rather than dragging them to the trash.

Best practices

1. Connect with an account that can see the whole schema, then verify object counts against the dictionary before reading any percentage.
2. Profile your NUMBER and DATE columns first and set the type mappings deliberately. Do this before the first conversion, not after the first load.
3. Script the conversion with the CLI so every run is identical and reviewable.
4. Sort red action items by object criticality, not by count. Twenty flags on a nightly batch job matter more than two hundred on a deprecated reporting view.
5. Write the semantic test suite before cutover: empty string versus NULL, date boundaries, numeric rounding, trigger ordering, transaction behavior in converted routines.
6. Instrument the target from day one. Query-level metrics in Amazon CloudWatch, or a Grafana stack against pg_stat_statements, will show you which extension pack wrappers are actually costing you before users complain.

Frequently asked questions

Is the AWS Schema Conversion Tool free?

The tool itself is a free download from AWS. What you pay for is everything around it: the compute you run it on, the target database, DMS replication instances for the data move, and the engineering time on the objects it hands back to you. That last item is the whole budget on most migrations.

Does it migrate data as well as schema?

No. It converts structure and code. Moving rows is AWS DMS, which handles both a full load and ongoing change capture. They are designed to be used together, and each has its own set of behaviors worth understanding separately.

Can I skip the extension pack entirely?

You can, but you are then hand-writing PostgreSQL equivalents for every Oracle built-in the pack would have covered. Configuring the tool to prefer orafce where possible is the usual compromise: fewer AWS-specific wrappers, open-source implementation, still automated. Full independence is achievable and is a refactoring project, not a setting.

Why did my report say 95% automatic and the project still take months?

Because the percentage counts objects, ignores runtime semantics entirely, and excludes application code, scheduled jobs, security, and downstream consumers. Tier two and the out-of-scope list in this post are where the months go.

Should I clean up the Oracle schema before or after conversion?

Before, every time. Dropping dead objects, unwrapping code you own, and removing synonyms on the source reduces the surface the tool has to reason about and shrinks the red list. It is also the only cleanup you can do while you still have a working system to test against.

How do I verify the conversion actually worked?

Row counts and DDL comparison are the floor, not the ceiling. Run the same business queries against both engines and compare results, with deliberate coverage of NULL handling, date boundaries, sorting and collation, and numeric rounding. Anything that only shows up under concurrency needs a load test, not a smoke test.

The one thing worth remembering

The AWS Schema Conversion Tool for Oracle is a good parser and an honest one about the things it cannot parse. The red list is trustworthy. The trouble is that a green tick means “I emitted valid PostgreSQL,” not “this behaves the way it did.” Dates lose their time, unqualified NUMBER lands as slow arithmetic, empty strings stop being NULL, triggers reorder themselves alphabetically, and none of that appears as a problem in the report.

Use it for what it is good at, which is the tedium you should never do by hand. Then spend your review time on the objects that passed, because that is where the expensive surprises live. Treat the assessment report as an inventory, never as a schedule.

Need a second pair of eyes on an Oracle to PostgreSQL conversion?

Most of the value in this work is in the review, not the click. Things I help with:

• Reading an assessment report properly: separating object counts from real effort, and re-scoping red items against the PostgreSQL version you are actually deploying.
• Auditing type mappings before you apply, with a column-level profile of NUMBER and DATE usage and a concrete mapping decision for each.
• Building the semantic test suite that catches empty-string and NULL divergence, trigger ordering, and rounding differences before users find them.
• Scripting the conversion through the SCT CLI so it is repeatable, reviewable, and lives in version control alongside the generated DDL.
• Planning the extension pack burn-down: which aws_oracle_ext calls are costing you, and what the native PostgreSQL replacement looks like.
• Mapping the out-of-scope surface, meaning scheduled jobs, grants, downstream reports and integrations, so cutover day has no surprises in it.

Send me an assessment report, a converted DDL file, or the action item list, and I will tell you what I would actually worry about in it.

The post AWS Schema Conversion Tool for Oracle: What Converts and What Quietly Doesn’t appeared first on John Nessime.

That argument settles architecture decisions all the time, and it’s the wrong argument. Both services return a primary-key lookup faster than your network hop and your ORM overhead. What hurts six months later is the query nobody wrote down at the start, and no benchmark will warn you about it.

So this post skips the latency charts. It covers the access pattern inventory you write before picking anything, where each service genuinely wins and genuinely hurts, a decision procedure you can run in an afternoon, and the arguments on both sides that fall apart on contact with a real workload.

Why the benchmark answers a question you don’t have

A benchmark measures a fixed operation under fixed conditions. Your application is not a fixed operation. It’s a set of reads and writes that grows every time somebody asks a new question about the data.

The failure mode that decides RDS vs DynamoDB is almost never throughput. It’s a product manager asking, eight months in, for every account that hasn’t logged in this quarter, broken down by plan tier. On Postgres that’s an awkward query and maybe an index. On DynamoDB, if you didn’t model for it, it’s a full table scan, a new global secondary index, a backfill, and a standing write-capacity tax to keep it in sync.

That asymmetry is the whole decision. A relational database defers the modeling cost and charges a little on every query for the flexibility. DynamoDB front-loads it: design keys around known patterns, get cheap predictable reads for exactly those, and pay dearly for anything else.

Write the access pattern inventory first

Before comparing anything, write the list. Every read and write the application performs, in plain language, with the shape of the operation and how fast it needs to be. This much is enough:

PATTERN                                SHAPE           FREQ    LATENCY
get order by id                        point read      high    p99 < 50ms
list a customer's orders, newest first range read      high    p99 < 100ms
orders by status, across customers     cross-entity    low     seconds fine
revenue by product category, monthly   aggregate       low     minutes fine
find orders by partial address         text match      medium  p99 < 300ms
mark order shipped + write audit row   multi-item write high   p99 < 100ms

Now read the SHAPE column. If nearly every row is a point or range read, and the exceptions tolerate latency, DynamoDB is a live option. If the column is full of cross-entity reads, aggregates and text matching, you’re describing a relational workload and can stop shopping.

The rows people leave out

The inventory is only useful if it’s honest, and the omissions are predictable:

• The admin back office. Support staff search by email, phone, partial name, anything. Ad hoc queries, and they never appear in the first inventory.
• Reporting. Someone will want a monthly rollup. Decide now whether it lives in the operational store or streams out to a warehouse.
• Deletion. Regulatory deletes cascade across entities. In a normalized schema that’s a transaction. In a denormalized single table it’s a fan-out you have to write and test yourself.
• Backfills. Every schema change becomes a migration job when no schema is enforced at the storage layer.

Write them down even if the answer is “we’ll handle it elsewhere.” A pattern you consciously exile to a warehouse is fine. One you forgot is a rewrite.

Where DynamoDB wins, and where it doesn’t

Give it its genuine case first, because it’s a strong one.

Where it wins

• Connection model. It’s an HTTPS API with SigV4 auth, so there’s no pool to exhaust. That’s why it pairs cleanly with Lambda. A relational database behind high Lambda concurrency needs RDS Proxy or a pooler, which is another component you operate.
• Predictable reads at any size. A query against a well-chosen partition key costs the same at ten thousand items or ten billion. Nothing in Postgres gives you that for free.
• Idle cost. On-demand mode bills per request unit consumed, so a table with no traffic costs storage and nothing else. Useful for tenant-per-table designs and low-traffic side services.
• Operational surface. No minor version upgrades, no vacuum, no failover window, no parameter groups to argue about.

The query API is narrow on purpose. Give it an exact partition key plus an optional sort key condition, and it returns an ordered slice of an item collection:

aws dynamodb query 
  --table-name AppData 
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" 
  --expression-attribute-values '{":pk":{"S":"CUSTOMER#42"},":sk":{"S":"ORDER#"}}'

One customer’s orders, in sort order, in one request, no join. If that call shape covers most of your inventory, you’re looking at the right service.

Where it doesn’t

• Anything you didn’t model. New patterns mean new keys or a new index, and a new global secondary index means a backfill plus write capacity on every future write.
• Filtering is not querying. A FilterExpression runs after the read, so you pay capacity for every item examined and only save bandwidth. A filter that discards most of what it reads is a design smell.
• Aggregates. There is no GROUP BY and no COUNT(*) over a table. You maintain aggregates yourself, usually with DynamoDB Streams feeding a Lambda that updates a counter item, which means you own the correctness problem when that consumer fails.
• Hard physical limits. An item caps at 400 KB. A Query or Scan page returns at most 1 MB before you paginate. A single partition sustains roughly 3,000 read and 1,000 write units per second, so a hot key throttles while the table sits well under its quota.
• Index budget. Global secondary indexes default to twenty per table; local secondary indexes cap at five and can only be defined at table creation. You can’t index your way out of a bad key design forever.

Where RDS wins, and where it doesn’t

Where it wins

• Unknown future queries. The entire point. Joins, aggregates, window functions, ad hoc analysis, and a new index in one statement when something turns out slow.
• Constraints the database enforces. Foreign keys, uniqueness, check constraints, transactions across arbitrary tables. In DynamoDB each of those is application code you write and maintain.
• Extensions. PostGIS for geospatial, pgvector for embeddings, full-text search built in, all against the same rows in the same transaction.
• Portability. Postgres and MySQL run on RDS, in a container, or on a plain VPS from a provider like Contabo or InterServer, so migrating off is an export and an import. DynamoDB’s data model doesn’t port without rewriting your data access layer.
• Everyone already knows SQL. Underrated on a small team. Teaching four developers single-table design costs real review time.

Where it doesn’t

• Connections under serverless concurrency. Each Lambda execution environment opens its own connection, and the database refuses new ones once max_connections is reached. RDS Proxy multiplexes to fix that, but some operations pin a client to a backend connection and pinned connections aren’t reused. Watch DatabaseConnectionsCurrentlySessionPinned before trusting the pool.
• Instances are always on. A provisioned instance bills whether or not anyone queries it. Aurora Serverless v2 changes that with a minimum of zero capacity units on a supported engine version, but resuming from paused is slow enough that the first user after an idle period feels it. AWS documents resume as taking roughly fifteen seconds.
• Vertical scaling has a ceiling. Writes go to one primary and read replicas help reads only. Sharding is a project, not a setting.
• Maintenance is yours. Version upgrades, failover testing, long transactions blocking vacuum, connection storms on restart. Managed does not mean absent.

If you want a database that hibernates properly, it lives in the scaling configuration and the idle timeout is yours to set:

aws rds create-db-cluster 
  --db-cluster-identifier app-cluster 
  --engine aurora-postgresql 
  --serverless-v2-scaling-configuration 
      MinCapacity=0,MaxCapacity=4,SecondsUntilAutoPause=600 
  --manage-master-user-password

A minimum of zero only works on engine versions supporting automatic pause and resume, so check yours before planning a budget around it. If the cluster refuses the setting, that’s usually why.

A decision procedure for RDS vs DynamoDB

Run these in order against the inventory you wrote. The first clear answer wins.

1. Can you enumerate the access patterns, and are they stable? If the product is still moving and nobody can say what the admin tools will need, pick the store that tolerates being wrong. That’s RDS.
2. Does any pattern need a cross-entity aggregate on the hot path? A user-facing screen needing a count or sum across entities in under a second, with no way to precompute it, means RDS.
3. Is there a natural partition key with even distribution? Tenant ID, user ID, device ID. If traffic concentrates on one key, the per-partition ceiling finds you regardless of capacity mode. No good key means no DynamoDB.
4. What’s the concurrency shape of the compute tier? Thousands of short-lived Lambda invocations argue for DynamoDB. A few long-running containers with a warm pool remove the strongest argument against RDS.
5. Does the workload go quiet? Bursty and mostly idle favors on-demand DynamoDB or a scale-to-zero Aurora cluster. Steady traffic makes provisioned capacity or a right-sized instance cheaper.
6. Who maintains this in a year? If it’s one part-time developer, weight familiarity heavily. A design nobody on the team can debug is a liability whatever the benchmark said.

The cost model nobody actually models

The two bill on axes that don’t compare. RDS charges instance hours, storage, and depending on engine and storage type, I/O, busy or not. DynamoDB charges request units and storage, so an idle table is nearly free and a chatty one isn’t.

The number that matters is request units per user action, and almost nobody calculates it. Reads bill in 4 KB increments and writes in 1 KB increments, strongly consistent reads cost twice what eventually consistent ones do, and transactional operations double again. Every index you project into consumes write units on every write that touches it. A screen firing six queries plus an audit write can cost many times the naive estimate.

Work it out per action, multiply by real traffic, and sanity check the instance side with my AWS cost estimator before the decision hardens. AWS recommends on-demand as the default capacity mode for most workloads, but steady predictable traffic is still cheaper provisioned, and you can flip between them:

# which mode is this table on?
aws dynamodb describe-table --table-name AppData 
  --query "Table.BillingModeSummary.BillingMode"

# what is it actually consuming?
aws cloudwatch get-metric-statistics 
  --namespace AWS/DynamoDB 
  --metric-name ConsumedWriteCapacityUnits 
  --dimensions Name=TableName,Value=AppData 
  --statistics Sum --period 3600 
  --start-time $(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)

Switching billing mode is rate limited, so treat it as a planned change rather than an incident-time knob.

Arguments that don’t survive contact

• “DynamoDB scales infinitely.” The service does. Your partition key might not. Uneven access concentrates load on one partition, and adaptive capacity only helps within limits.
• “Postgres can’t handle our scale.” Usually said before anyone has run EXPLAIN (ANALYZE, BUFFERS) on the slow query. A missing index and an N+1 in the ORM explain most claims of outgrowing relational.
• “We’ll add the indexes later.” True in Postgres, expensive in DynamoDB. A new global secondary index backfills across the whole table and then taxes every subsequent write.
• “Serverless database means no operations.” You trade instance operations for capacity and key-design operations. Throttling, hot partitions and stream consumer lag all need dashboards, exactly like a relational instance does.
• “We can just use both.” Sometimes right, often a way to avoid deciding. Two stores means two consistency models, two restore procedures, and a sync path that becomes the least reliable component you own.

How I’d decide

My default for a new application with an unfinished product definition is managed Postgres. Not because it’s better, but because being wrong about a relational schema costs an afternoon and being wrong about a key design costs a migration.

I’d reach for DynamoDB first when the workload is a small closed set of patterns keyed off one identifier, with high write volume, tolerance for eventual consistency, and concurrency that would flatten a connection pool. Session stores, telemetry ingestion, per-user feeds, idempotency keys, feature flags. It’s excellent at all of those.

The hybrid worth doing is narrow: relational as the system of record, DynamoDB for one high-volume pattern that joins nothing. The hybrid to avoid splits related entities across both stores and reconciles them in application code.

Whichever you land on, instrument it on day one. Throttled requests and consumed capacity on the DynamoDB side, connection counts and slow queries on the relational side, on a CloudWatch or Grafana Cloud dashboard someone actually opens. If you’re restricting item-level access with IAM condition keys such as dynamodb:LeadingKeys, run the policy through my IAM policy checker before it ships.

Frequently asked questions

Is DynamoDB always faster than RDS?

For a primary-key lookup at large scale its latency is more consistent, because that operation’s cost doesn’t grow with table size. For a query it wasn’t modeled for it can be far slower, because the fallback is a scan. Speed depends on the match between query and key design, not on the service.

Can I run analytics on DynamoDB?

Not directly, and you shouldn’t try. Export the table to S3 or stream changes out, then query with Athena or load a warehouse. Treat operational and analytical stores as separate systems.

What about Aurora Serverless as the middle ground?

It solves idle cost, not connections or write scaling. If you’re considering DynamoDB because of Lambda concurrency or single-primary write limits, Aurora Serverless doesn’t address that. If it’s because you’re paying for an idle dev database, it does.

How do I migrate from DynamoDB to RDS if I get it wrong?

Export to S3, reshape denormalized items into normalized tables, load, and dual-write during cutover. The data movement is routine. The hard part is that your application code assumed a key-value access shape everywhere, so the rewrite lands in the service layer, not the pipeline.

Is DynamoDB cheaper than RDS for a small application?

Usually yes when traffic is low and bursty, because on-demand charges per request against an idle table while an instance bills continuously. The advantage narrows as traffic steadies and reverses once throughput justifies provisioned capacity or a reserved instance.

Do I need RDS Proxy if I use Lambda with Postgres?

At meaningful concurrency, yes. Below that, a direct connection reused across invocations is fine, and the proxy adds cost plus a pinning failure mode. Add it when connection counts approach your limit, not preemptively in development.

Conclusion

The one thing worth remembering about RDS vs DynamoDB: you’re not choosing a performance profile, you’re choosing when to pay the modeling cost. DynamoDB takes it up front and pays you back in predictability. A relational database lets you defer it and charges interest later in query complexity.

Write the access pattern inventory. Be honest about the admin screens and the reporting nobody has asked for yet. If the list is short, stable and keyed on one identifier, DynamoDB will serve you for years. If it isn’t, take the relational database and spend the saved effort on indexes and connection handling.

Need a second opinion on the data store decision?

Cheap to review now, expensive to revisit later. Things I help with:

• Building the access pattern inventory with your team and pressure-testing it against the queries your roadmap implies
• Reviewing a DynamoDB key and index design before it ships, including partition key distribution and index write amplification
• Diagnosing throttling and hot partitions on an existing table, and whether the fix is a key change, a capacity change or a cache
• Sizing RDS or Aurora for a serverless compute tier, including connection handling, RDS Proxy configuration and pinning
• Modeling the cost of both options against your real request volume instead of a rule of thumb
• Planning a migration in either direction, with a dual-write cutover and a rollback that works

Send me your access pattern list, a slow query plan, or your throttling metrics and I’ll tell you what I’d do with it.

The post RDS vs DynamoDB: The Benchmark Isn’t the Decision appeared first on John Nessime.

Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.

That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.

Why external documents break pipelines that passed testing

Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:

• Container. One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.
• Text origin. Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.
• Layout stability. The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.

You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.

Failure family one: the file is out of spec before Textract sees it

Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.

Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.

Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. BadDocumentException means Textract could not read the file, and retrying fails identically. DocumentTooLargeException means route to the asynchronous path or split the file. InvalidS3ObjectException is usually IAM or a bucket-region mismatch, not a document problem. Only InternalServerErrorException deserves a retry with backoff.

Failure family two: synchronous versus asynchronous is a correctness gate

This is the one that silently eats data, so it goes early.

The synchronous operations, DetectDocumentText and AnalyzeDocument, cap at 10 MB in memory. For PDF and TIFF input they also cap at one page. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.

The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.

A synchronous call looks like this. The document lives in S3, and you name the features you want:

aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1

The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:

import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]

Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:

blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break

Check JobStatus before you trust Blocks. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.

Failure family three: the response is a graph, not a record

Textract does not return fields. It returns a flat list of Block objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.

For form data, the shape is specific. A key-value pair is two blocks of type KEY_VALUE_SET, distinguished by their EntityTypes array containing either KEY or VALUE. The key block carries a relationship of type VALUE holding the ID of its value block, and a relationship of type CHILD holding the IDs of the WORD blocks that spell out the label. The value block carries only CHILD relationships pointing at its own words. So reading one field means three hops.

blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])

Two things about that snippet. The SELECTION_ELEMENT branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.

Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.

Before hand-rolling any of this, look at the AWS-maintained parsers. The amazon-textract-response-parser package and the higher-level amazon-textract-textractor library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.

Getting Amazon Textract structured data out of the right API

Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting TABLES on a corpus with no tables costs money and returns noise.

• DetectDocumentText. Raw text and geometry. Use it when you only need a search index or a text layer.
• AnalyzeDocument with FORMS. Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.
• AnalyzeDocument with TABLES. Grid reconstruction with row and column indices. The right call for line-item grids and statements.
• AnalyzeDocument with QUERIES. You ask natural language questions and get answers back. This is the fix for layout drift.
• AnalyzeDocument with LAYOUT and SIGNATURES. Reading order and structural elements; signature locations for compliance checks.
• AnalyzeExpense. Purpose-built for invoices and receipts, with normalized field types.
• AnalyzeID. Purpose-built for identity documents.

Why Queries beats FORMS on documents you don’t control

FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor’s wording. When they change “P.O. No.” to “Order Reference”, your parser looks for a key that no longer exists and writes a null.

Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:

aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1

The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.

Invoices are their own problem

For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each ExpenseDocument holds SummaryFields for header-level data and LineItemGroups containing the line items. Fields carry a normalized Type such as VENDOR_NAME, plus a ValueDetection and an optional LabelDetection.

That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.

Failure family four: confidence is per field and it is not your risk model

Every extracted element carries a confidence score. Two mistakes follow from that.

The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.

The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:

1. Auto-accept. Above threshold and passing a format check. Straight into the target table.
2. Review. Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.
3. Reject. Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.

Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.

Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.

Failure family five: reprocessing you didn’t budget for

Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline’s life, because that is what parsers do.

So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.

import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"

Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.

One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&O SafeErase exist for exactly that gap between “moved to trash” and “actually gone”.

Where Textract stops being the right tool

Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.

Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.

The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.

Troubleshooting

• Only the first page came through. A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.
• Output truncates partway through a long document. Missing continuation token handling on the get operation. Loop until the token is absent.
• Fields visible on the page never appear in output. The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.
• Quality collapsed for one sender only. Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.
• Throttling during a backfill. Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.
• Table cells land in the wrong columns. Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.

Common mistakes

• Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.
• Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.
• Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.
• Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.
• Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.

Best practices

• Classify before you extract. Route each document to the narrowest API that covers it.
• Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.
• Store raw responses immutably, keyed by content hash, and parse from storage.
• Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.
• Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.
• Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.
• Keep a golden set of real documents with known correct answers, and run it on every parser change.

Frequently asked questions

Can Amazon Textract process multi-page PDFs?

Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.

What is the difference between FORMS and Queries?

FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.

How much does Amazon Textract cost?

Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.

Does Textract handle handwriting?

It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.

Which languages does Amazon Textract support?

Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.

Should I use Textract or Bedrock Data Automation?

Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.

Conclusion

If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.

So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.

Need help turning documents into a pipeline you can trust?

Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:

• Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.
• Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.
• Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.
• Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.
• Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.
• Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.

If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.

The post Turning External Documents Into Structured Data With Amazon Textract appeared first on John Nessime.