That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.

This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.

SageMaker Unified Studio is a front door, not a runtime

Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.

AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project’s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new authorisation layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.

What it actually replaces

The pile of standalone studios and query editors

This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.

The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a “workspace” meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.

Per-team domain sprawl

The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.

The access request spreadsheet

There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.

The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.

What it doesn’t replace

Your bill, or your ability to explain it

There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.

What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer’s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.

Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.

Your IAM and Lake Formation design

Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.

Nobody in the adoption meeting will say “this fixes our permissions mess.” Plenty of people will quietly assume it. Say it out loud early.

Your infrastructure as code

This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.

It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.

SageMaker AI

The original SageMaker was renamed SageMaker AI and remains available on its own. If your team’s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.

What SageMaker Unified Studio quietly adds

Here is the section that rarely makes it into the migration plan.

You now operate DataZone, intentionally or not

SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a dzd prefix, and the console for creating one still lives under the DataZone console path.

You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:

# Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID

Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.

Project creation is a CloudFormation deploy wearing a friendly hat

Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.

This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation “just doesn’t work,” do not reproduce it in the UI. Read the stack events:

# Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table

The first failed resource causes the rollback. Everything after it is noise. This one habit turns most “the platform is broken” tickets into five-minute answers.

A domain-type fork you will live with for a long time

Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.

Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post’s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.

Where adoption actually goes wrong

“It works in my console but not in the project”

This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.

Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:

aws sts get-caller-identity

If the ARN is not the one you assumed, stop. You have found the problem.

The bill went up after a migration that was supposed to save money

Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.

Nobody can find the data that is definitely there

Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.

Common mistakes

• Treating it as a migration rather than an addition. Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.
• Letting users self-serve projects on day one. Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.
• Choosing the domain authentication type in a hurry. It is the hardest thing here to change later.
• Skipping the catalog. Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.
• Assuming the CI/CD CLI provisions your platform. It deploys applications into projects that already exist.
• Not tagging project resources from the start. Retrofitting attribution across dozens of blueprint-created resources is miserable work.

How I’d decide

A rough order of operations for anyone weighing this up:

1. Ask what problem you are solving. If it is “our teams cannot discover or safely share each other’s data,” this is the right tool. If it is “we have too many consoles,” it is an expensive way to buy fewer bookmarks.
2. Count your teams. The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.
3. Decide the authentication model first. Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.
4. Write the project profiles yourself. The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.
5. Prove the teardown path before the build path. Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.
6. Keep the old consoles. They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.

One more piece of judgement: wire your existing observability in from the start rather than relying on the studio’s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.

Frequently asked questions

Does SageMaker Unified Studio replace SageMaker Studio?

It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.

Do I still need Amazon DataZone separately?

Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.

How much does SageMaker Unified Studio cost?

The interface has no standalone charge. You pay for each AWS service you use through it on that service’s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.

Can I manage domains and projects with Terraform?

AWS’s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.

What happens to my Glue jobs and Athena queries?

Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.

Is it worth adopting for a small team?

Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.

Can I run both the old and new experiences at once?

Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.

The one thing worth remembering

SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.

That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was “fewer tabs.” The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.

Ask which one you are before the first project gets created, not after the fortieth.

Need a second opinion on your adoption plan?

This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:

• Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup
• Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults
• Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use
• Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover
• Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants
• Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long

Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.

The post SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t appeared first on John Nessime.

The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.

I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.

This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.

The bill arrives before the carousel does

Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.

Solution training. New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.

Inference capacity. A campaign has minProvisionedTPS, a domain recommender has minRecommendationRequestsPerSecond. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.

Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.

import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])

Decision one: domain recommender or custom solution

A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.

The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.

Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.

Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.

On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.

One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at aws-hrnn, its advice about cold items and training cadence is stale too.

Decision two: how interaction data gets in

Bulk imports replace unless you tell them not to

This is the one that costs people a month of confused model quality.

A dataset import job defaults to FULL, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is INCREMENTAL, which appends and only replaces records sharing an ID.

Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.

import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)

Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.

Streaming events through an event tracker

The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the personalize-events client.

import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)

Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.

That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.

The sessionId field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.

Decision three: schema and event types, decided once

Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.

{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}

The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.

Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.

Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.

On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.

Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.

Decision four: freshness without a retraining bill

Two mechanisms get conflated constantly, and the difference sets your monthly cost.

Automatic training creates a new solution version. The model learns from your latest data. It costs a training run each time.

Automatic updates are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.

So automatic updates solve the cold item problem and do nothing about model drift. AWS’s own guidance is that even with them, training cadence should stay at least weekly.

Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.

The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.

Decision five: how the recommendation reaches the page

Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.

Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.

Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.

Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.

Troubleshooting an AWS ecommerce recommendation pipeline

• Everyone gets the same recommendations. Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.
• New products never show up. Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.
• Training fails on data volume. Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.
• The dataset shrank and nothing errored. A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.
• Recommendations ignore what the user just clicked. Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.
• The bill rose during a quiet week. Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.
• Filters are not excluding purchased items. Filtering looks at a bounded slice of each user’s history per event type. Heavy users push older purchases outside it.

Auditing an account you did not build starts with listing what exists, not with reading the application code.

aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN

Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.

Common mistakes

• Leaving automatic training on for a solution that exists only to satisfy curiosity.
• Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.
• Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.
• Recording events only for authenticated users, starving the model of most of your traffic.
• Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.
• Calling PutEvents synchronously on the render path.
• Following an implementation guide that still references the retired HRNN recipes.
• Shipping with no fallback, so the first slow inference call becomes a slow product page.

Best practices

• Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.
• Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.
• Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.
• Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.
• Prefer batch inference plus a cache for anything that does not have to react within the session.
• Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.
• Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.

Frequently asked questions

How much data do I need before Personalize is worth it?

The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.

Can I build this without Amazon Personalize?

Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.

What does an idle Personalize setup cost?

More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.

Real-time or batch recommendations?

Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.

Do new products appear automatically?

Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.

How do I connect a Shopify or WooCommerce store?

There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.

How often should I retrain?

At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.

The one thing worth remembering

An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.

Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.

Need a second pair of eyes on your recommendation pipeline?

This is the kind of work I do for ecommerce teams building on AWS:

• Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.
• Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.
• Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.
• Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.
• Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.
• Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.

If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.

The post Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite appeared first on John Nessime.

The uncomfortable answer is that people did know. The superintendent knew the drywall crew had been short for two weeks. The project engineer knew a structural RFI had been sitting with the design team well past its due date. The foreman knew the weekly commitments kept slipping. None of it was hidden. It just never reached one screen, in one place, while there was still float left to spend.

This post walks through building a construction early warning dashboard that catches those signals while they are still cheap to act on. It covers why schedule delay shows up late, the one data decision that makes early warning possible, the signals worth tracking, how to pull data out of Primavera P6, Microsoft Project and Procore, and how to alert without training everyone to ignore the alerts.

Why Construction Delay Shows Up Three Weeks Late

Delay rarely arrives as one event. It accumulates in small amounts across activities that do not look critical yet. Three things keep it invisible.

1. The schedule is updated in batches. If the CPM schedule is statused monthly, or even biweekly, the finish date is only recalculated at that cadence. Everything that happened in between lands at once.
2. Progress is self-reported. Percent complete is an estimate from whoever statuses the activity. Optimism is not dishonesty, but it compounds. An activity that sits at a comfortable-looking percentage for three updates is telling you something.
3. The data is overwritten in place. This is the one that bites hardest. Most schedule tools, and most dashboards built on top of them, show the current value of float, the current forecast finish and the current status. Last month’s values are gone unless someone saved the file.

That third point is why so many project dashboards look fine right up until they don’t. A current-state dashboard tells you where you are. Delay warnings live in the rate of change. An activity with twelve days of float looks healthy. The same activity with twelve days of float, down from thirty over the last two updates, is the next critical path.

The Design Decision That Makes Early Warning Possible: Snapshot Everything

Before any chart or alert, decide that every schedule update, every RFI log pull and every weekly plan gets stored as a dated snapshot. You append. You never overwrite. Storage for this is trivial; a few hundred activities per update is nothing for any database.

PostgreSQL is what I reach for first here. It runs happily on a small managed instance like Amazon RDS or DigitalOcean Managed Databases, or on a modest VPS from Hetzner, Contabo or Interserver if you are comfortable handling backups yourself. A minimal activity snapshot table looks like this:

CREATE TABLE activity_snapshot (
    project_code     text        NOT NULL,
    data_date        date        NOT NULL,
    task_code        text        NOT NULL,
    task_name        text,
    total_float_days numeric,
    early_start      date,
    early_finish     date,
    is_complete      boolean     NOT NULL DEFAULT false,
    loaded_at        timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_code, data_date, task_code)
);

The primary key does the important work. Each activity appears once per data date, so loading the same export twice fails loudly instead of silently doubling the history. The data_date is the schedule’s status date, not the day you loaded the file, because trends must be plotted against when the schedule says it was true.

Keying on task_code, the activity ID the planner sees, rather than an internal database ID matters too. Internal IDs can change when a schedule is re-imported. Activity IDs usually survive, and when they don’t, you want to know (more on that in troubleshooting).

Early Warning Signals That Move Before the Finish Date Does

Each of these is a leading indicator of schedule delay. None is perfect alone. Together they give you weeks of notice instead of a surprise in the monthly update.

Float Erosion on Near-Critical Paths

Watching the critical path is necessary but late. By the time an activity is critical, its float is already gone. The useful signal is near-critical work losing float quickly. This query compares each open activity to its own previous snapshot:

WITH history AS (
    SELECT
        project_code,
        task_code,
        task_name,
        data_date,
        total_float_days,
        LAG(total_float_days) OVER (
            PARTITION BY project_code, task_code
            ORDER BY data_date
        ) AS prev_float_days
    FROM activity_snapshot
    WHERE NOT is_complete
)
SELECT
    project_code,
    task_code,
    task_name,
    prev_float_days,
    total_float_days,
    prev_float_days - total_float_days AS float_lost_days
FROM history
WHERE data_date = (SELECT max(data_date) FROM activity_snapshot)
  AND prev_float_days IS NOT NULL
  AND total_float_days <= 20
  AND prev_float_days - total_float_days >= 5
ORDER BY float_lost_days DESC;

The thresholds (twenty days remaining, five days lost) are starting points, not rules. Tune them to your update cadence and project length. What matters is that the query looks at movement, which a current-state report cannot.

Constraint Aging: RFIs and Submittals Against Need-By Dates

An overdue RFI on its own is noise. Every project has them. An overdue RFI that blocks an activity starting in ten days is a delay you can see coming.

The hard part is the link. RFI and submittal logs in Procore or Autodesk Construction Cloud rarely point at schedule activity IDs. Someone has to maintain a small mapping table from RFI or submittal to activity, or at least to area and spec section. I would not skip it. Without the link you get a list of late paperwork; with it you get “this answer is needed in eight days and the ball has been in the architect’s court for three weeks.”

The signal to chart is days of slack between the expected response and the need-by date of the linked activity. When it goes negative, the RFI is now on the schedule’s path whether the schedule knows it or not.

Commitment Reliability: PPC and Reasons for Variance

If the field runs the Last Planner System, you already have a strong signal. Percent Plan Complete (PPC) is the share of weekly commitments finished as promised. Each week’s PPC is a lagging number, but the trend across weeks leads the CPM schedule, because missed promises show up in the field long before they show up as recalculated dates.

Store reasons for variance as categories, not free text. A PPC dip caused by weather tells you something different from three straight weeks of “material not on site.” The second one is a procurement problem that will keep producing delay until someone fixes it.

Look-Ahead Churn

Count how many times an activity’s planned start has moved across snapshots of the three- or six-week look-ahead. An activity pushed out once is normal planning. One that has been pushed four updates running is being deferred, not planned, and the reason is usually a constraint nobody has named yet.

Earned Value, Read Carefully

If you track earned value, the Schedule Performance Index (SPI = EV / PV) belongs on the dashboard, with a caveat many dashboards skip. Classic SPI drifts back toward 1.0 as the project nears completion, even when the job is late, because earned value and planned value both end at the budget at completion. A late-stage SPI that looks like recovery may just be arithmetic.

Earned Schedule fixes this by measuring in time rather than money. It asks at what point in the baseline your current earned value was supposed to be reached, then divides that by the actual time elapsed to get SPI(t). It needs no new data collection, only different math on the same numbers. If you only plot one schedule index late in a job, plot SPI(t).

Claimed Progress Versus Manpower

Daily logs record who was on site. When an activity’s reported percent complete keeps rising while the manpower logged against that trade stays well below the plan, one of those numbers is wrong. Usually it is the percent complete. This comparison is crude, and it needs a consistent trade mapping, but it is one of the few checks that uses data nobody is trying to make look good.

Getting the Data Out of P6, Microsoft Project and Procore

A construction project dashboard lives or dies on boring extraction work. Plan for it.

Primavera P6 XER Exports

An XER file is plain tab-delimited text. Lines beginning with %T name a table, %F lists its fields, and %R lines are rows. Activities live in the TASK table. This reader pulls one table into a list of dictionaries:

def read_xer_table(path, table_name):
    rows, fields, current = [], None, None
    # XER exports often use a Windows code page, not UTF-8
    with open(path, encoding="cp1252", errors="replace") as fh:
        for line in fh:
            parts = line.rstrip("rn").split("t")
            tag = parts[0]
            if tag == "%T":
                current = parts[1]
            elif tag == "%F" and current == table_name:
                fields = parts[1:]
            elif tag == "%R" and current == table_name:
                rows.append(dict(zip(fields, parts[1:])))
    return rows

tasks = read_xer_table("update.xer", "TASK")
for t in tasks[:5]:
    print(t["task_code"], t["task_name"], t["total_float_hr_cnt"])

Two details catch people. Float and durations in XER are stored in hours (total_float_hr_cnt), so converting to days means dividing by the hours per day of each activity’s calendar, not a blanket eight. And an activity with a value in act_end_date has an actual finish, which is a clean way to set is_complete without relying on status labels.

Use the float P6 stored in the export rather than recomputing the critical path yourself. P6’s result reflects its scheduling options, constraints and calendars. Your recompute probably won’t.

Microsoft Project

Microsoft Project can save a schedule as XML, which carries tasks, dates and total slack. The same snapshot logic applies. Save one file per status date, load it with its data date, and never replace the previous one.

Procore and Other Project Management Platforms

Procore’s REST API exposes RFIs and submittals per project, including status, due dates and ball-in-court. Two warnings. Most useful endpoints are scoped to a single project, so a portfolio view means iterating over the project list. And submittal status values can be configured per company, so map them explicitly rather than hard-coding what you saw on one account. Autodesk Construction Cloud has its own RFI API with a similar shape.

The Spreadsheet Nobody Wants to Admit Exists

The weekly work plan is very often an Excel file or a Smartsheet. Accept that. A scheduled job that reads a fixed-layout sheet from a shared folder is less elegant than an API and far more likely to get used. Fighting the field’s tools is how early warning systems die.

Building the Dashboard and Alerts People Don’t Mute

For visualization I lean toward Grafana, self-hosted or on Grafana Cloud, because its alert rules can run SQL directly against PostgreSQL and route results to email, Slack or Microsoft Teams. You can see the style of dashboard in this live Grafana demo.

Power BI deserves a fair hearing. Many contractors already license it, project controls teams know it, and its reporting and drill-down are excellent for monthly reviews. Where it is weaker is rule-based alerting on arbitrary queries. If your organization lives in Microsoft 365, a sensible split is Power BI for the reporting pack and a small scheduled job or Grafana instance for the alerts. Metabase is another reasonable middle ground for smaller teams.

Whatever the tool, the alert design matters more than the charts:

• Send a weekly digest, not a page. Schedule risk is not an outage. A Monday summary of every signal that crossed a threshold gets read. Twenty individual notifications get filtered.
• Name the owner in the alert. “RFI 142 blocks level 3 slab pour, need-by in 8 days, ball in court: structural engineer” is actionable. “Constraint risk: high” is not.
• Compare the project to itself. Portfolio-wide thresholds punish complex jobs and let simple ones coast. A drop from its own recent average is the better trigger.
• Alert on stale data. If no schedule update has loaded in longer than your cadence, that is itself a warning. An early warning dashboard showing old data is worse than none.

Troubleshooting

Float values look wrong compared to P6

Check the hours-to-days conversion first. Then check which float definition the schedule uses. A project-level “must finish by” date or activity constraints can produce negative float that looks like a bug but is correct. Also confirm whether the planner defines critical as longest path or as float below a threshold.

Trend lines break after a re-baseline or renumbering

If activity IDs change, LAG() finds no previous row and the activity appears brand new. Add a check that counts activities present in the previous snapshot but missing from the current one. A sudden jump means the history needs a mapping table, not a fresh start.

PPC sits at or near 100 percent every week

That usually means commitments are being made at the end of the week, or tasks are so loose they cannot fail. A PPC that never moves carries no signal. Look at how commitments are recorded before trusting the number.

The dashboard is green but the field says you are behind

Check how percent complete is calculated. P6 supports duration, physical and units percent complete types, and they can tell very different stories about the same activity. Believe the field, then find which input is out of step.

Common Mistakes

• Building on the live schedule only, with no history, so every trend has to be reconstructed by hand.
• Watching only the critical path and missing the near-critical work that becomes critical next month.
• Trusting classic SPI in the final third of a project, when it drifts toward 1.0 regardless of performance.
• Tracking overdue RFIs without linking them to the activities they block.
• Recomputing float yourself instead of using what the scheduling tool calculated.
• Sending so many alerts that the project team creates an inbox rule to hide them.

Best Practices

• Append every export with its data date. Never overwrite history.
• Start with two signals, float erosion and constraint aging, and add more once those are trusted.
• Agree with the planner on update cadence. The dashboard cannot be fresher than its inputs.
• Keep the RFI-to-activity mapping small and owned by one named person.
• Review thresholds after the first month. Early numbers are guesses.
• Show each signal with its source and data date, so nobody argues about whether it is current.

Frequently Asked Questions

What are leading indicators of construction delay?

The most useful are float erosion on near-critical activities, RFIs and submittals aging toward the start of the work they block, a falling PPC trend, activities repeatedly pushed out of the look-ahead, and reported progress outpacing logged manpower.

How often should the schedule be updated for early warning to work?

More often than monthly if you can manage it. Weekly or biweekly updates give the trend queries enough points to show movement. Signals from RFIs and weekly plans can refresh daily regardless of schedule cadence.

Is SPI a good early warning metric?

Early and mid-project, yes, alongside float and critical path analysis. Late in a project, classic SPI drifts toward 1.0 even on late jobs, so the time-based SPI(t) from Earned Schedule is more trustworthy.

Can I build this with Power BI instead of Grafana?

Yes for the reporting side, and it is often the better choice where the team already uses it. The snapshot database and trend logic are the same either way. Rule-based alerting is where you may want a separate tool.

Do I need Primavera P6 for a construction early warning dashboard?

No. Any scheduling tool that exports activities with dates and float works, including Microsoft Project. What you need is a consistent export on every update and somewhere to keep the history.

How do I link RFIs to schedule activities?

Usually with a small mapping table maintained by the project engineer: RFI or submittal number, activity ID, and need-by date. Mapping by area and spec section is a workable first step if activity-level links are too much at the start.

Conclusion

The project manager who learns about delay three weeks late usually isn’t short of data. The problem is that the data is spread across a schedule, an RFI log and a weekly plan, and each one only remembers its latest state. A construction early warning dashboard works because it keeps the history and watches the movement: float draining from work that isn’t critical yet, answers arriving after the work that needs them, promises slipping week after week. Snapshot first. Everything else is a query on top.

Need Help Building a Construction Early Warning Dashboard?

I build and host the data side of this for construction teams and project controls groups. Here is where I can help:

• Setting up a snapshot database that stores every P6 or Microsoft Project update with its data date.
• Writing XER and XML loaders that handle hours-based float, calendars and renumbered activities.
• Pulling RFIs and submittals from Procore or Autodesk Construction Cloud and linking them to activities.
• Building Grafana dashboards for float erosion, constraint aging, PPC and SPI(t) trends.
• Designing weekly digest alerts with named owners, sent to email, Slack or Teams.
• Hosting and maintaining the dashboard so your team reads it instead of running it.

If you want to scope it first, the Engagement Builder lays out phases and effort. Or send me one recent XER export and your RFI log, and I will tell you which signals your data can already support.

The post The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard appeared first on John Nessime.

No. And that question is the entire problem with this one.

This post is a reference for the PHP2Shell vulnerability: what actually broke inside PHP, which servers are exposed, how to patch each common setup, how to prove the fix is really running, and which quick fixes give you false comfort. Keep it open while you work through your servers.

What happened: the PHP2Shell vulnerability in plain terms

PHP2Shell is the nickname researchers gave to two flaws in the PHP engine itself, CVE-2026-109618 and CVE-2026-109619. PHP rates both 9.2 on CVSS v4.0, and both can let an unauthenticated attacker run code on a web server. The name is the researchers’, not the PHP project’s.

The dates matter here, because they decide if you were exposed and for how long:

• 20 October 2026: PHP announced an emergency release for two critical issues. Details leaked publicly the same day.
• 21 October 2026: fixed releases shipped: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, with a full advisory.
• 23 October 2026: PHP published an informational advisory for a third, much narrower issue, CVE-2026-115560, with a one-line mitigation and no emergency release.

The flaws were reported by Wiz Research, Calif and OpenAI’s Codex Security team, who used AI models to turn long-standing weaknesses into working exploits. The affected code goes back to PHP 5.4, so effectively every PHP version from the last decade is in scope.

The two bugs, without the jargon

CVE-2026-109618 (out-of-memory corruption). PHP enforces memory_limit by aborting a request the instant it crosses the limit. If that abort lands while PHP is still building internal request data, for example while parsing a multipart file upload, the data is left half-initialized. Request cleanup then treats it as valid and works with memory that can still hold attacker-controlled leftovers.

CVE-2026-109619 (stale global state). Some state owned by one request can survive after its memory has been freed, for example after a fatal error. The next request handled by the same worker can use that stale state before normal setup resets it.

Here is why this is worse than the usual plugin CVE. The bug lives in the runtime, not in your code. A site whose own code is flawless is still exposed, and updating WordPress, Laravel or Drupal changes nothing, because the PHP binary underneath comes from your OS packages, your host or your container image. The researchers also demonstrated reading memory from the PHP worker process, which can contain passwords, tokens and keys belonging to other people’s requests, without ever getting a shell.

Who is exposed, and why mod_php is the worst case

PHP’s advisory covers every installation that serves web requests: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. The risk is not equal across them, though.

mod_php is the highest risk. Apache with mod_php runs request shutdown even when request startup already failed. That means simply running out of memory while PHP parses the request body is enough, and one unauthenticated request can trigger the attack before a single line of your application runs. PHP demonstrated this with default memory and input limits.

PHP-FPM is harder to hit, not safe. FPM shuts the worker down when a request fails early in setup, which blocks the cleanest path. Public reporting says one of the two bugs was still shown to reach code execution under FPM, through a path that depends on something the application does. Treat “we run FPM” as “we have slightly more time,” nothing more.

If you are not sure which mode you run, these two checks answer it on most Linux servers. The first lists Apache modules and shows a PHP module only if mod_php is loaded. The second lists running FPM services.

apachectl -M | grep -i php
systemctl list-units --type=service | grep -i fpm

How to fix PHP2Shell: it depends on where your PHP comes from

The fix is always the same in principle: run a patched PHP build. Getting there depends entirely on who builds your PHP. And there is a trap that runs in both directions. Linux distributions and long-term-support vendors received the patch early and backported it, so a patched package can carry an old-looking version like 8.3.6. Meanwhile, an unpatched process can keep running after the package updated. The version string answers neither question reliably.

Debian and Ubuntu packages

Debian, Ubuntu and Ondřej Surý’s widely used repository (packages.sury.org, also published as ppa:ondrej/php) all shipped fixed packages, including builds for some end-of-life branches. Update normally, then confirm the CVE is in the package changelog instead of reading the version number. Replace 8.3 with your branch.

sudo apt update && sudo apt upgrade
zgrep CVE-2026-109618 /usr/share/doc/php8.3-common/changelog.Debian.gz
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2

If zgrep prints a line, the fix is in the installed package. If it prints nothing, you are either on an older distro release that needs a paid extended-support channel, or on a repository that has not shipped the fix.

RHEL, AlmaLinux and Rocky Linux

Same idea with RPM. Update, then search the package changelog for the CVE. If you use Remi’s parallel packages, the package name follows its own pattern (for example php83-php-common), so query that name instead.

sudo dnf upgrade 'php*'
rpm -q --changelog php-common | grep CVE-2026-109618
sudo systemctl restart php-fpm httpd

One warning from PHP’s own advisory: the volunteer-run rpms.remirepo.net repository was not expected to have updates on release day, and PHP suggested switching to distribution packages or building from source if you depend on it. Check before assuming the update landed.

cPanel, CloudLinux and LiteSpeed

cPanel ships its own ea-php packages through EasyApache 4, and CloudLinux ships alt-php. CloudLinux backported the fix into supported and end-of-life branches, so most of its fixed builds keep version numbers below the php.net release. CloudLinux’s published procedure is to update both package families, then restart the services that run PHP so no worker keeps the old binary loaded.

yum clean all && yum update 'alt-php*' 'ea-php*' -y
/scripts/restartsrv_httpd

If your domains run PHP-FPM through cPanel, restart the FPM service as well. On LiteSpeed, restart the web server so every lsphp process respawns from the new build. Public analysis says LiteSpeed’s LSAPI behaves more like FPM than mod_php, which buys time but does not remove the need to patch.

DirectAdmin with CustomBuild

CustomBuild compiles PHP from source, so you get the fix when CustomBuild offers the patched release and you rebuild. Refresh the version list, confirm the fixed release appears for each PHP version you run, then build.

cd /usr/local/directadmin/custombuild
./build update
./build versions
./build update_versions

Read the ./build versions output before running the last command. If a branch you use still shows an unpatched release, apply the mitigations below for that branch until it updates. Compiling takes a while on small servers, so plan for it rather than starting it at peak traffic.

Docker and container images

A container never patches itself. If your image is built FROM an official php tag, rebuild with a fresh pull of the base image and redeploy. If your Dockerfile pins the base by digest, the digest has to change or nothing happens.

docker compose build --pull
docker compose up -d

Then run php -v inside the running container, not on the host, to confirm the version the app actually uses.

Managed hosting versus unmanaged VPS

On managed hosting, the host owns the PHP runtime, and several large hosts announced they had patched their fleets. Your job there is to get written confirmation, not to assume. On an unmanaged VPS from Contabo, Interserver or any similar provider, nobody patches PHP but you. That is the trade you accepted for the lower price and full root access, and this week is when it gets collected.

End-of-life PHP: 8.1 and older

The PHP project released no fixed builds below 8.2. Because of the severity, its security team published minimal source backports for PHP 7.4, 8.0 and 8.1 for people who build from source. Beyond that, your options are a distribution or vendor that backports (Ubuntu’s own PHP 8.1 on 22.04, the Surý repository, Freexian ELTS for older Debian, CloudLinux and TuxCare, HeroDevs), or upgrading. Note that PHP 8.2 itself reaches end of life on 31 December 2026, so if you are touching an 8.2 server now, plan the next upgrade at the same time.

Prove the patch is actually running

This is the failure mode that will bite most people. The package manager says “upgraded,” the ticket gets closed, and the old PHP code keeps serving traffic because the processes that loaded it were never restarted. Linux happily keeps running a binary whose file has been replaced on disk.

1. Confirm the fix is in the package with the changelog check above, not the version string.
2. Restart everything that runs PHP: every FPM pool, Apache (for mod_php), LiteSpeed, and any long-running PHP workers such as queue consumers managed by Supervisor or systemd.
3. Check the restart time is after the upgrade. systemctl status php8.3-fpm shows when the service started. If that is earlier than your package install, it is still running old code.
4. Ask the system what still needs a restart. On Debian and Ubuntu, sudo needrestart -r l lists services still using outdated files, without restarting anything.
5. Check the right binary. php -v on the command line reports the CLI build, which can differ from the FPM build serving your site. On Debian and Ubuntu, php-fpm8.3 -v asks the FPM binary directly.

Mitigations if you cannot patch today, and what they don’t do

PHP is blunt about this: only an update is a complete fix. The settings below reduce exposure to the specific attack paths it has confirmed. They do not make a server safe, and some of them have real costs. Use them to buy hours, not weeks.

Set them where applications cannot override them: php_admin_value or php_admin_flag in the FPM pool, or the Apache configuration for mod_php. Do not rely on ini_set() or an FPM .user.ini file, because those apply too late in the request. Here is the FPM pool version on a Debian-style path (adjust the file and branch to yours):

; /etc/php/8.3/fpm/pool.d/www.conf
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 6M
php_admin_flag[session.upload_progress.enabled] = off

And the Apache equivalent for mod_php, inside the virtual host or a Directory block:

php_admin_value memory_limit -1
php_admin_value post_max_size 8M
php_admin_value upload_max_filesize 6M
php_admin_flag session.upload_progress.enabled off

Restart every worker afterwards. Now the reasoning behind each line, because each one protects a different path:

• memory_limit = -1 targets CVE-2026-109618. The confirmed exploit needs a predictable per-request limit to hit, so removing the limit removes the trigger. The cost: one request can now consume all available memory. A genuine allocation failure can also still reach the vulnerable code.
• Low post_max_size shrinks the request bodies an attacker can use. PHP suggests 8M where uploads are not needed and 32M where they are, with upload_max_filesize kept below it.
• session.upload_progress.enabled = off targets CVE-2026-109619. It blocks the confirmed session paths but not other variants. It does nothing at all for CVE-2026-109618.
• On PHP 8.5 only, fatal_error_backtraces = off blocks the confirmed fatal-backtrace path for CVE-2026-109619, and nothing else.

Because removing the memory limit opens a denial-of-service risk, PHP recommends a container or cgroup memory limit as the availability boundary, ideally one that kills only the affected worker. It specifically warns against RLIMIT_AS or RLIMIT_DATA, since those make allocations fail at the limit, which is exactly the condition the bug needs. Enforcing request-size, field-count and multipart-part limits at your reverse proxy helps too.

A web application firewall is a reasonable stopgap, not a fix. Cloudflare deployed rules before the patch shipped and reported attackers changing payloads to get around them. If your site already sits behind Cloudflare, those rules help. They do not replace the update.

The third issue: CVE-2026-115560

Two days after the release, PHP disclosed one more possible code-execution path. It needs an unusual setup: the application sets session.serialize_handler to php_binary or php_serialize in configuration, then switches it to php at runtime with ini_set() before calling session_start(), with session upload progress still enabled. The fixed releases above do not cover it. PHP’s advice is to keep session.upload_progress.enabled=0 until the next regular bugfix release. If you already set it for CVE-2026-109619, leave it in place and you are done.

After patching: assume someone tried

Amazon reported a honeypot hit against these flaws on 17 October, four days before disclosure, and exploitation attempts climbed fast once details were public. Most reports so far describe scanning and failed attempts rather than confirmed break-ins, but “nobody has reported it” is not evidence about your server. If you ran an unpatched, internet-facing PHP site during that window, do three things.

• Rotate secrets a PHP worker could have held in memory: database passwords, API keys, payment provider keys, SMTP credentials and, for WordPress, the salts in wp-config.php (which logs every user out, so warn people first).
• Read your logs for the patterns researchers flagged: unusual multipart requests followed by memory-limit errors or bursts of HTTP 500 responses, oddly large PHPSESSID cookies in responses, and PHP workers restarting with no explanation.
• Look for anything new and executable in your web roots and upload directories, especially PHP files that no deploy put there.

Troubleshooting

“php -v still shows the old version.” Either it is a distro backport (check the changelog for the CVE), or you are looking at the CLI binary while a different build serves the site. Check the FPM binary directly and the container, not the host.

“The site throws 500 errors after the update.” The fix changes how PHP handles running out of memory. Crossing memory_limit now raises a soft error at a safe point, and a second, higher threshold ends the process outright, without running shutdown handlers. A plugin or import job that used to squeak by can now fail differently. Check the error log, then review memory_limit and post_max_size against what the application genuinely needs.

“Cleanup or logging code stopped running on big jobs.” Same cause. If anything important lives in a shutdown function, it will not run when the hard limit is hit. Fix the memory usage rather than raising limits blindly.

“The package manager says there is nothing to update.” You are probably on a repository that has not shipped the fix, an end-of-life distro release, or an end-of-life PHP branch with no vendor backport. That is a migration problem, not a patching problem.

“I set the mitigation but the old value still shows.” Look for overrides in other pool files, per-directory configuration or application code, and make sure you used the admin variants and restarted the workers.

Common mistakes

• Updating WordPress, plugins or the framework and calling it done. None of that touches the PHP runtime.
• Trusting the version number in either direction, and declaring a backported package “vulnerable” or a stale process “patched.”
• Upgrading the package and never restarting FPM, Apache or long-running workers.
• Treating session.upload_progress.enabled=0 as the fix. It was circulated as a workaround before the release and only covers part of one CVE.
• Setting mitigations with ini_set(), .htaccess overrides that apps can change, or .user.ini, which take effect too late.
• Leaving memory_limit=-1 in place long after patching, with no cgroup boundary behind it.
• Forgetting the boxes nobody owns: the staging server, the old client site on PHP 7.4, the forgotten container in a side project.

Best practices

• Keep an inventory of every PHP install, its source (distro, panel, container, source build) and its branch. This week shows why.
• Verify fixes by CVE in the package changelog, and verify the running process by start time.
• Patch mod_php servers first, then public FPM sites, then internal tools.
• Once patched, restore a sane memory_limit, but keep session upload progress off until PHP’s next regular release covers CVE-2026-115560.
• Move servers off mod_php to PHP-FPM where you can. It would not have saved you here, but it narrowed the easiest path.
• Stay on a supported PHP branch, or pay for a vendor that backports. An unsupported branch turns every emergency into a migration.

FAQ

What is the PHP2Shell vulnerability?

It is the nickname for two memory-safety flaws in the PHP engine, CVE-2026-109618 and CVE-2026-109619, both rated 9.2 on CVSS v4.0. They can allow unauthenticated remote code execution on servers that run PHP for web requests.

Which PHP versions fix PHP2Shell?

PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 from php.net. Distribution and vendor packages may carry the fix under different, older-looking version numbers, so check the package changelog for the CVE.

Is my WordPress site affected?

If it runs on an unpatched PHP build, yes, regardless of how up to date WordPress and its plugins are. The flaw is in PHP itself, so the fix comes from your server, host or container image.

Is disabling session.upload_progress enough?

No. It blocks the confirmed session paths for one of the two bugs and covers the narrow third issue, CVE-2026-115560. It does nothing for CVE-2026-109618. Only an update is a complete fix.

I’m on PHP 7.4 or 8.1. What now?

php.net has no release for you, but PHP published source backports for 7.4, 8.0 and 8.1, and several vendors and distributions shipped backported packages. Use one of those immediately, then plan the upgrade to a supported branch.

Do I need to restart anything after updating PHP?

Yes. Restart every PHP-FPM pool, Apache if you use mod_php, LiteSpeed, and long-running PHP workers. Until they restart, they keep running the vulnerable code already loaded in memory.

Should I rotate passwords and keys?

If an internet-facing server ran an unpatched build after details went public, it is a sensible precaution. Researchers showed worker memory could be read remotely, and that memory can hold credentials from other requests.

Conclusion

The PHP2Shell vulnerability lives below your application, so the only real fix is a patched PHP build that is actually running. Everything else in this post is either a way to get there or a way to buy time. If you remember one thing, make it this: verify the fix by CVE in the package, and verify the process by its restart time. The version string will mislead you in both directions.

Need help patching PHP2Shell across your servers?

If you have more PHP servers than certainty, I can help you close this out properly:

• Inventory every PHP install across your VPS, cPanel, DirectAdmin and container hosts, with branch and package source.
• Patch each one through the right channel and prove the fix by CVE and process restart time.
• Apply and later roll back the interim mitigations safely, with a memory boundary in place.
• Rebuild and redeploy PHP container images, including pinned-digest pipelines.
• Review logs for the published indicators and plan secret rotation without breaking production.
• Plan the move off end-of-life PHP branches and off mod_php.

Send me the output of php -v, your package changelog check, and how PHP runs on the server (FPM, mod_php or LiteSpeed), and I’ll tell you where you stand.

The post PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked appeared first on John Nessime.

That is the failure mode that makes this comparison worth writing about. The Parquet vs JSON vs CSV question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.

This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else’s benchmark, and an honest look at where CSV and JSON still beat the columnar option.

The billing model is the whole story

Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.

Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery’s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.

That last part catches people. LIMIT 10 does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.

So the question is not “which format is smaller on disk.” It is “how few bytes can the engine get away with reading.” Those diverge fast.

Four levers, not one

When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.

1. Column pruning

Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.

2. Block skipping

Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.

The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group’s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.

3. Splittability

This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.

Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.

If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.

4. Request overhead

Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine’s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.

CSV: still the right answer more often than people admit

Where it wins. Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.

Where it doesn’t. No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.

CSV is an excellent transport format and a poor storage format. That distinction is the useful one.

JSON: the most expensive way to store a table

Where it wins. Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.

Where it doesn’t. Every record repeats every key name. Store ten million events with a field called customer_identifier and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.

JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.

Parquet: cheap reads, real obligations

Where it wins. Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.

Where it doesn’t. This is the part the enthusiastic posts skip.

• You cannot append. Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.
• Schema mismatches across files are a runtime error, not a warning. Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.
• Timestamps are a compatibility minefield. The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.
• It is opaque. You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.
• Writing costs more. Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.

None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.

Measure it on your data, in about twenty minutes

Published benchmarks tell you about somebody else’s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.

1. Get a real byte count for what you have now. Not an estimate, the actual prefix.

aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable

The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.

2. Convert a representative slice, not the whole thing. One partition is enough. In Athena, a CTAS statement writes the output for you.

CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';

Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.

3. Run your three most common queries against both tables and compare bytes scanned. Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.
4. On BigQuery, price it before you run it. A dry run costs nothing and returns the byte estimate.

bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'

5. Inspect the Parquet file you just wrote. Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.

python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"

That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.

6. Put a guardrail up before anyone else touches it. Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.

Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.

Arguments that don’t survive contact

• “We gzip our CSVs, so we already save the money.” Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.
• “Parquet is always faster.” Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.
• “We’ll just convert everything to Parquet.” Converting a landing zone that receives appended events all day means fighting the format’s biggest weakness. Land raw, convert on a schedule, query the converted copy.
• “Storage is cheap so format doesn’t matter.” Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.
• “Our BI tool caches, so nobody re-scans.” Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.

How I’d decide

Not a scorecard. A short sequence of questions, in order.

• Is this data leaving your organisation? CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.
• Is it arriving continuously, one record at a time? Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.
• Is it read more than once, by more than one person, with filters? Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.
• Is the whole dataset smaller than a few hundred megabytes? Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.
• Are you fighting append semantics, concurrent writers, or schema evolution? You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.

The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.

Frequently asked questions

How much smaller is Parquet than CSV?

It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.

Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?

Less than you would think, because once data is loaded into a BigQuery native table it lives in Google’s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.

Is ORC better than Parquet?

They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet’s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.

Why did my query cost the same after I converted to Parquet?

Usually one of three things. You are running SELECT *, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.

Should I use Snappy, GZIP or ZSTD for Parquet?

Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.

Can I query CSV and Parquet in the same table?

Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.

What about the cost of the conversion job itself?

Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.

The one thing worth remembering

The Parquet vs JSON vs CSV decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.

Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.

Need help cutting a data lake bill?

Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:

• Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves
• Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses
• Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control
• Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges
• Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types
• Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end

If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.

The post Parquet vs JSON vs CSV: What Each Format Actually Costs You appeared first on John Nessime.

The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.

That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.

What managed compaction actually does

S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.

Compaction is the Iceberg rewriteDataFiles routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table’s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.

There are four strategies. auto is the default and picks for you: sort order defined in metadata means sort, otherwise binpack. binpack packs small files into bigger ones. sort orders rows hierarchically by your chosen columns so engines prune more files. z-order interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.

Both sort and z-order need a sort_order in your Iceberg table properties and the s3tables:GetTableData permission to set. They also cost more per byte processed than binpack, because they do more work.

Failure family one: maintenance that fails without telling anyone

This is the one worth reading the post for. Maintenance jobs have four states: Successful, Failed, Disabled, and Not_Yet_Run. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.

aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table

The response carries a block for each of icebergCompaction, icebergSnapshotManagement, and icebergUnreferencedFileRemoval, each with its own status, a lastRunTimestamp, and a failureMessage. The three are independent. Compaction reporting Successful tells you nothing about whether the other two are running.

The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.

Tags and branches disable snapshot expiry for the whole table

If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.

Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month’s bill rather than in their terminal.

Iceberg retention properties conflict with the managed config

If history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in metadata.json. It refuses instead.

This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.

ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');

gc.enabled set to false

Setting the Iceberg gc.enabled table property to false stops both snapshot management and unreferenced file removal. Leave it unset or explicitly true. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.

KMS key policy missing the maintenance principal

If the table is encrypted with a customer managed key, that key’s policy has to grant access to the service principal maintenance.s3tables.amazonaws.com. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.

Failure family two: the storage tail you keep paying for

Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.

When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from minSnapshotsToKeep (default 1) and maxSnapshotAgeHours (default 120). Both conditions must be satisfied before a snapshot goes.

When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for nonCurrentDays (default 10), then are deleted permanently.

Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through unreferencedDays (default 3) into the same noncurrent window.

Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.

# Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'

Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.

Failure family three: write amplification and the compaction charge

S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.

The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.

People reach for targetFileSizeMB first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.

The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.

Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.

Failure family four: files compaction quietly skips

Compaction supports Parquet, Avro, and ORC. It does not support the Fixed data type, nor the brotli or lz4 compression codecs.

That last one matters because lz4 is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.

Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the write.format.default table property to avro or orc. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.

Failure family five: compaction and your writers fighting over the table

Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.

When compaction loses you see Failed with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between Failed and Successful.

When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.

Turning managed compaction off, and when that is right

Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.

The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.

Disabling is a per-table change, and it does not require zeroing the settings.

aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'

From there you run maintenance yourself. Trino exposes an optimize procedure through ALTER TABLE ... EXECUTE, and Spark has the standard Iceberg rewrite_data_files procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.

Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.

Troubleshooting S3 Tables compaction and maintenance

Work through these in order. The first step resolves more cases than the rest combined.

1. Check all three job statuses, not one. Run get-table-maintenance-job-status and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.
2. Read the failure message literally. It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means history.expire.* is set.
3. Status is Disabled or Not_Yet_Run unexpectedly. Someone replaced the configuration. Confirm with get-table-maintenance-configuration, and remember put-table-maintenance-configuration replaces rather than merges.
4. Compaction runs but file count stays high. Check codec and data types against the unsupported list. Files using brotli or lz4, or containing the Fixed type, are outside what compaction touches.
5. Insufficient-access failures on an encrypted table. Check the KMS key policy grants the maintenance.s3tables.amazonaws.com service principal, and that the key is enabled and still exists.
6. Storage still growing with everything green. Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.

For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer “when did this start” rather than “what is broken now”.

Common mistakes

• Treating the three maintenance jobs as one thing. They fail independently and for different reasons. “Maintenance is enabled” is not a status.
• Creating branches or tags on a managed table. A normal Iceberg workflow that switches off snapshot expiry for the whole table.
• Tuning targetFileSizeMB to reduce cost. It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.
• Setting sort or z-order without a sort order defined. The strategy needs sort_order in table properties and the s3tables:GetTableData permission, and it costs more per byte than binpack.
• Running write pipelines with no commit retry. Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.

Best practices for running S3 Tables compaction in production

• Poll job status on a schedule and alert on it. A small scheduled function calling get-table-maintenance-job-status across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on Failed and on a stale lastRunTimestamp, on whatever dashboard you already watch, Grafana or CloudWatch.
• Make maintenance config part of your IaC. These are settings someone can change with one CLI call and no review. Codify them so drift is visible.
• Set retention windows from a real requirement. Ask how far back anyone needs to time travel, then set maxSnapshotAgeHours and nonCurrentDays to that plus a margin, rather than accepting defaults by omission.
• Batch on the write side before tuning anything else. Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.
• Choose the strategy from the query pattern. One dominant filter column suggests sort. Several equally weighted predicates suggest z-order. Neither suggests leaving it on auto.
• Track the compaction billing dimension separately. It moves independently of storage and it is the earliest signal that a writer’s behaviour has changed.

Frequently asked questions

How do I know if S3 Tables compaction is actually running?

Call get-table-maintenance-job-status and read the icebergCompaction block. It returns Successful, Failed, Disabled, or Not_Yet_Run, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.

Why is my S3 Tables storage growing when compaction says it succeeded?

Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.

Can I run my own compaction alongside the managed one?

You can, but it is a poor idea. Both are writers competing under Iceberg’s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.

Does disabling compaction stop all S3 Tables charges?

No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.

Is sort or z-order compaction worth the extra cost?

It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour sort. Several dimensions in varying combinations favour z-order. Queries that mostly scan everything justify neither.

Can I recover data deleted by S3 Tables maintenance?

Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.

The one thing worth remembering

S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.

What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on get-table-maintenance-job-status that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.

Need help with Iceberg tables on AWS?

I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:

• Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots
• Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana
• Working out where a bucket’s storage growth actually comes from, and setting retention windows against a real requirement
• Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them
• Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick
• Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance

If something is not adding up, send me the output of get-table-maintenance-job-status for the table and a rough description of your write pattern. That is usually enough to say what is going on.

The post S3 Tables Compaction in Production: The Cost of Convenience appeared first on John Nessime.

That is a different question, and most evidence buckets fail it quietly. Versioning is on, the bucket is encrypted, access logging is enabled, and there is still an admin role in the account that can call DeleteObjectVersion on every object in it. Nothing is wrong until someone asks you to prove a negative.

An audit evidence repository on AWS is not a bucket with a good name. It is a write path, a retention decision you cannot take back, and a set of guardrails that survive the person who built them leaving the company. This post covers how to build one with S3 Object Lock and AWS Backup Vault Lock, and more usefully, the five ways I have seen the design go wrong in ways nobody notices until the evidence is needed or the bill arrives.

What “immutable” actually buys you on AWS

Three separate mechanisms get called immutability, and they protect different things.

• S3 Object Lock stops an object version from being overwritten or deleted for a defined period. It works only on versioned buckets, and once you enable it on a bucket you cannot disable it or suspend versioning again.
• AWS Backup Vault Lock does the same job for recovery points in a backup vault, with its own retention floor and ceiling enforced at the vault level.
• Integrity validation, such as CloudTrail’s digest files, proves a file has not changed since delivery. That is a separate property from being undeletable, and auditors increasingly ask for both.

Object Lock is the core of an audit evidence repository on AWS because evidence usually arrives as files: exported reports, signed approvals, config snapshots, log archives. Vault Lock matters when the evidence is a backup of a database or an EBS volume. Integrity validation is what turns “we kept it” into “we kept it and here is the hash chain”.

Failure one: picking the retention mode you cannot walk back

Object Lock has two retention modes and the difference is not cosmetic. In governance mode, a caller who holds the s3:BypassGovernanceRetention permission can shorten or remove retention and delete the version. In compliance mode, nobody can, including the account root user, and the retention period cannot be shortened or downgraded to governance.

Read that second sentence again with a typo in mind. Set seven years where you meant seven days, in compliance mode, and the only exit AWS documents is closing the account. That is not a bug. It is the entire point of the mode. But it means the mode is a business decision wearing a technical costume.

How I decide: compliance mode only where an external rule names a retention period and an external party will inspect it. Everything else starts in governance mode, with the bypass permission granted to nobody by default and denied outright in a service control policy. Governance mode with no principal holding the bypass right is functionally immutable for day-to-day purposes, and it leaves you a documented, auditable escape hatch that requires an explicit policy change to use.

The bypass is not silent, either. Overriding governance retention requires the x-amz-bypass-governance-retention:true header on the request, which means the action shows up in CloudTrail as a distinctly shaped call. That is a good alarm to build.

# Create the bucket with Object Lock on. Versioning is enabled automatically
# and cannot be suspended afterwards.
aws s3api create-bucket 
  --bucket my-evidence-archive 
  --object-lock-enabled-for-bucket 
  --region eu-west-1 
  --create-bucket-configuration LocationConstraint=eu-west-1

# Default retention for NEW objects written to the bucket.
aws s3api put-object-lock-configuration 
  --bucket my-evidence-archive 
  --object-lock-configuration 
  '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

Start there. Run it for a quarter. Watch what actually lands in the bucket and how big it gets. Then decide whether compliance mode is worth its permanence.

Failure two: retention that was never applied to the objects that matter

This is the invisible one, and it is the reason to read the rest of this post.

Object Lock is enabled at the bucket level, but retention lives on the individual object version. A default retention rule applies to objects written after the rule exists. It does not reach backwards. Enable Object Lock on an existing bucket holding three years of evidence and you get a bucket that reports Object Lock as enabled while every object already in it carries no retention at all.

The bucket console looks correct. The compliance checklist gets ticked. The old evidence is still deletable by anyone with delete permissions.

Two things to do about it. First, backfill retention on existing versions, either object by object with put-object-retention or, for anything above a few thousand keys, with S3 Batch Operations, which exists precisely for applying retention at scale. Second, verify per object rather than per bucket.

# Check what protection a specific version actually carries.
# Look for ObjectLockMode, ObjectLockRetainUntilDate, ObjectLockLegalHoldStatus.
aws s3api head-object 
  --bucket my-evidence-archive 
  --key 2024/q3/access-review-export.csv 
  --version-id "SOME_VERSION_ID"

The stronger fix is to stop trusting the writer to set retention correctly and enforce a floor in the bucket policy. S3 exposes the s3:object-lock-remaining-retention-days condition key for exactly this, so you can reject writes that ask for too short a lock.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RejectShortRetention",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-evidence-archive/*",
      "Condition": {
        "NumericLessThan": {
          "s3:object-lock-remaining-retention-days": "365"
        }
      }
    }
  ]
}

The write path now fails loudly when someone gets it wrong, which is what you want. A rejected upload gets noticed. A silently unprotected object does not.

Failure three: the delete that looks like a delete

Object Lock protects object versions. It does not stop new versions being created, and it does not stop delete markers being placed on top of a protected object.

So a user with ordinary delete permissions can issue a delete against a locked object, get a success response, and make it vanish from every normal listing. The protected version is still there. Nothing was destroyed. But an auditor browsing the bucket sees nothing, and the person who ran the command believes the file is gone.

Three practical consequences:

• Deny s3:DeleteObject on the evidence prefix for everyone except a break-glass role. Immutability is not a substitute for permissions, it is a backstop under them.
• Any retrieval tooling you hand to an auditor must list object versions, not objects. list-object-versions, not list-objects-v2.
• Lifecycle rules interact with this in a way that surprises people. An expiration rule on a versioned bucket adds a delete marker rather than removing data, and a rule that tries to permanently expire a noncurrent version will not remove a version still under retention. Your cleanup policy and your retention policy are two different clocks, and the lock always wins.

Failure four: evidence that arrives already unprovable

Immutability starts when the object lands. Everything before that is chain of custody, and it is where most repositories are weakest.

If the evidence is CloudTrail, use the integrity validation CloudTrail already ships. With log file validation enabled, CloudTrail writes an hourly digest file containing hashes of the log files delivered in that hour, signs each digest with a per-region private key, and puts the digests in a separate folder in the same bucket. You can then verify the chain later.

# Turn on digest files for an existing trail.
aws cloudtrail update-trail 
  --name org-audit-trail 
  --enable-log-file-validation

# Verify a time range later. Files must still be in the location
# CloudTrail delivered them to, or validation cannot run.
aws cloudtrail validate-logs 
  --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-audit-trail 
  --start-time 2020-01-01T00:00:00Z

Two details worth knowing before you design around it. Validation only covers log files referenced by digest files, so a gap in digest coverage is a gap in your proof. And the files must stay where CloudTrail put them, which means copying logs into a tidier prefix breaks the built-in verification and hands you the job of writing your own checker.

For evidence that is not CloudTrail, the equivalent is doing the work yourself: hash the artifact at the point of generation, record the hash somewhere separate from the artifact, and write both. A manifest per collection run, checksummed and locked alongside the files, is unglamorous and holds up.

One constraint that catches people mid-build: a bucket with Object Lock enabled cannot be used as the destination for S3 server access logs. If your design had one bucket receiving both the evidence and its own access logs, it needs splitting.

Failure five: the bill you cannot switch off

Compliance mode plus a long retention period plus archival storage classes is where cost surprises live, because every lever you would normally pull to reduce spend has been deliberately disabled.

The mechanics that drive it, none of which are secret but all of which get discovered late:

• Minimum storage durations. S3 Glacier Flexible Retrieval carries a 90 day minimum and S3 Glacier Deep Archive carries 180 days. Transition earlier than that and you are billed for the remainder of the minimum anyway.
• Per-object overhead in Glacier classes. AWS documents 40 KB of metadata overhead per object for Glacier Flexible Retrieval and Deep Archive, of which 8 KB is billed at S3 Standard rates and 32 KB at the destination rate. For a repository full of 2 KB JSON approvals, the overhead is the bill.
• Billing starts when the rule is satisfied. Lifecycle transitions to Glacier classes happen asynchronously, but you are charged at the destination rate, and the minimum duration clock starts, from the date the rule matched rather than the date the bytes physically moved.
• You cannot delete your way out. Under compliance mode, oversized or mistakenly retained objects stay until their retain-until date passes, whatever the finance conversation says.

The design response is to aggregate before you lock. Batch small artifacts into per-day or per-collection archives, checksum the archive, and lock the archive rather than thousands of tiny files. It costs you a little retrieval convenience and saves you a per-object overhead multiplier you cannot renegotiate later.

Also pick the retrieval tier against the real deadline. Deep Archive is the cheapest place to keep evidence and one of the slowest to get it back. If a regulator or a legal team may need files at short notice, that trade is a bad one regardless of the storage line item.

Putting the repository together

With the failure families in view, the build order is fairly short.

1. Separate the account. Evidence lives in its own AWS account, not a prefix in the production log bucket. The blast radius of a compromised production admin should not include the record of what they did.
2. Write the service control policies first. At the organisation level, deny s3:BypassGovernanceRetention, deny s3:PutBucketPolicy and s3:DeleteBucketPolicy on the evidence bucket, and deny disabling the trail. SCPs apply to the account root too, which is the point.
3. Create the bucket with Object Lock and a default governance retention. Block public access, enforce TLS with an aws:SecureTransport condition, and encrypt with a customer managed KMS key whose policy is written for this bucket alone.
4. Enforce the retention floor in the bucket policy using the remaining-retention-days condition, so a misconfigured collector fails rather than writes something unprotected.
5. Replicate to a second account or region. Object Lock works with S3 Replication, and the destination bucket must also have Object Lock enabled. This is your answer to region-level loss and to an account-level mistake.
6. Add Vault Lock for backup-shaped evidence. Snapshots and database backups belong in a locked backup vault rather than being exported to S3 by hand.
7. Alarm on the write path. A day with no new evidence objects is the failure that never pages anyone. Alert on absence, not just on errors.

Vault Lock deserves its own note because its compliance mode has a grace period that Object Lock does not. You choose a cooling-off window, which AWS requires to be at least three days, during which the lock can still be removed. After that the vault and its lock are immutable and cannot be changed or deleted by anyone, including AWS.

# Compliance mode: the presence of --changeable-for-days is what makes it
# compliance rather than governance. Minimum value is 3 (72 hours).
aws backup put-backup-vault-lock-configuration 
  --backup-vault-name evidence-vault 
  --min-retention-days 365 
  --max-retention-days 2555 
  --changeable-for-days 3

Use that window. Run a backup job through the vault, confirm the retention arithmetic against what the policy document actually requires, and only then let the clock run out.

One more design question worth asking out loud: does every copy need to be inside AWS? A second, offline-ish copy on independent infrastructure answers a different threat than cross-region replication does, because it survives an account-level compromise. Providers like InterServer, Contabo and Hetzner sell cheap storage that can hold an encrypted archive, and several S3-compatible services implement Object Lock semantics of their own. Confirm the provider actually supports Object Lock rather than assuming an S3-compatible API implies it, because that varies.

Troubleshooting

AccessDenied when deleting a version you are allowed to delete

Usually governance retention plus a missing bypass. The IAM permission s3:BypassGovernanceRetention and the explicit bypass on the request are both required. If the object is in compliance mode, there is no combination of permissions that will work, and the error is telling you the truth.

# Only valid for governance mode, and only with the matching IAM permission.
aws s3api delete-object 
  --bucket my-evidence-archive 
  --key path/to/object 
  --version-id "SOME_VERSION_ID" 
  --bypass-governance-retention

Replication fails on an Object Lock bucket

The destination needs Object Lock enabled too. Replicated objects take the lock configuration of the source, so a destination without Object Lock cannot accept them. Check the destination bucket first, then the replication role’s permissions.

Lifecycle rule appears to do nothing

Two likely causes. The versions it targets are still under retention, so expiration cannot remove them. Or the rule targets current versions on a versioned bucket, in which case it is adding delete markers and the data is still there as noncurrent versions, still being billed.

validate-logs reports invalid or missing files

Confirm nothing moved the objects. Validation requires the log and digest files to be in the location CloudTrail delivered them to, and the command runs per region. A copy job that reorganised the prefix layout is the usual culprit.

Object Lock cannot be enabled on the bucket

Versioning has to be enabled first, and Object Lock cannot be turned off later. If the bucket is currently a server access log destination, that has to move before Object Lock will work for you at all.

Common mistakes

• Enabling Object Lock on an existing bucket and assuming the objects already in it are now protected. They are not, until retention is applied to each version.
• Choosing compliance mode on day one to look thorough, then discovering the retention period was wrong.
• Treating Object Lock as an access control. It is a floor under your permissions, not a replacement for them.
• Leaving the bypass permission attached to a general administrator role because it was needed once during testing.
• Locking millions of tiny objects into Glacier classes and paying per-object overhead on all of them.
• Storing evidence in the same account as the systems it describes, which makes the evidence only as trustworthy as that account’s admins.
• Never testing retrieval, so the first restore attempt happens under audit pressure with an unfamiliar Glacier restore workflow.

Best practices for an audit evidence repository on AWS

• Governance mode by default, compliance mode only where an external requirement names the period. Write down which rule drove the choice, in the same repository.
• Enforce the retention floor in the bucket policy so the write path fails loudly instead of writing unprotected objects.
• Use legal hold, not extended retention, for investigations. Legal holds have no expiry and are removed explicitly, which is the right shape for something that ends on an unknown date.
• Aggregate small artifacts into archives with a checksummed manifest before locking them.
• Replicate to a second account with Object Lock enabled, and check replication metrics rather than assuming.
• Rehearse retrieval quarterly, including the Glacier restore step, and keep the output as evidence of the rehearsal.
• Monitor for absence of writes. Tools like Amazon CloudWatch, Grafana or Datadog are all fine here. The alert that matters is “no evidence arrived today”.

Frequently asked questions

Can I enable S3 Object Lock on a bucket that already exists?

Yes. S3 supports enabling Object Lock on existing buckets, and versioning must be enabled. The important caveat is that enabling it does not retroactively protect the objects already stored. You apply retention to those separately, per object or with S3 Batch Operations.

What is the difference between governance mode and compliance mode?

Governance mode can be overridden by a principal holding s3:BypassGovernanceRetention who sends the bypass header. Compliance mode cannot be overridden by anyone, including the root user, and the retention period cannot be shortened or the mode downgraded.

Does S3 Object Lock stop someone deleting the file?

It stops the protected version being destroyed. It does not stop a delete marker being placed on top, which hides the object from ordinary listings while the version remains. Keep delete permissions tight and list versions when you audit.

Do I need Object Lock if I already have versioning and MFA delete?

They solve different problems. Versioning preserves history but a sufficiently privileged caller can still remove versions. Object Lock removes that possibility for the duration of the retention period, which is the property an auditor is asking about.

How do I prove the evidence has not been modified?

For CloudTrail, enable log file integrity validation and keep the signed digest files with the logs, then run aws cloudtrail validate-logs for the period in question. For other artifacts, hash at generation time, store the manifest separately, and lock both.

Can a lifecycle rule delete objects that are still locked?

No. Retention wins over lifecycle expiration. Plan the two together, because a cleanup rule that quietly does nothing for years is a cost problem you will not see until you look at storage growth.

Is Vault Lock compliance mode reversible?

Only during the cooling-off window you set at creation, which must be at least three days. After the lock date, the vault and its lock cannot be changed or deleted by anyone, including AWS.

The one thing worth remembering

An audit evidence repository on AWS fails in a specific, quiet way: the bucket says Object Lock is enabled, and the objects inside carry no retention. Everything else in this post is downstream of that gap. Check protection at the object version, not at the bucket, and enforce the floor in the write path so the system tells you when it is wrong.

Then pick the mode like the one-way door it is. Governance mode with a locked-down bypass gets you almost all of the protection and keeps a documented way back. Compliance mode is worth its permanence when a real external rule demands it, and expensive theatre when it does not.

Need help building or reviewing one of these?

Immutable retention is easy to enable and easy to get subtly wrong, and the mistakes are the kind you find out about at the worst moment. This is work I take on directly:

• Designing an evidence account with Object Lock, KMS, service control policies and a locked-down write path
• Auditing an existing bucket to find object versions that carry no retention despite Object Lock being enabled
• Backfilling retention across existing objects with S3 Batch Operations without locking in the wrong period
• Modelling the long-run cost of a retention decision before compliance mode makes it permanent
• Setting up cross-account replication, AWS Backup Vault Lock and CloudTrail integrity validation as one coherent design
• Writing the retrieval runbook and rehearsing it, so the first restore is not during an audit

If you want a second opinion on what you already have, send me the bucket policy, the Object Lock configuration and the output of a head-object call on one old evidence file. That is usually enough to tell you where the gap is.

The post Building an Audit Evidence Repository on AWS: Immutable Retention Without Regret appeared first on John Nessime.

So you sketch the obvious thing. Amazon Cognito for sign-in, Amazon S3 for the documents, a small front end in between. It’s three services and a weekend, and it works on the first try, which is exactly the problem. A secure patient document portal built this way passes every test you think to run, because the tests you think to run are all “can Alice download Alice’s file?” The one that matters is “what can Alice see about Bob?” and that one has an answer you won’t like.

This post covers the parts that go wrong after the demo. Where to put the authorization boundary, how to scope the S3 policy so one patient can’t enumerate another, why your download links expire earlier than you configured, what encryption actually gates, and what your audit trail looks like on the day someone asks for it. Everything here is mechanism, not a compliance checklist. Getting a Business Associate Agreement in place is a separate conversation with your lawyer and your AWS account team.

The failure that never pages you

Here’s the shape of it. You write an IAM policy that lets a signed-in user read objects under their own prefix. Something like arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*. You test it. Alice gets her file. Alice cannot get Bob’s file. Ship it.

Then, because the front end needs to render a file list, someone adds s3:ListBucket. And s3:ListBucket is not an object-level action. It applies to the bucket ARN, not the object ARN, so the ${cognito-identity.amazonaws.com:sub} substitution in your resource string does nothing for it. Unless you add a separate condition on the request, a signed-in patient can list every key in the bucket.

They still can’t download anything. Every GetObject outside their prefix returns a denial. But they can read the key names. And key names in a clinic bucket are rarely neutral. patients/8f2a.../oncology-referral-2024.pdf tells you something about a person even if the bytes stay locked.

Nothing alerts on this. No error appears in any log you’re watching, because nothing errored. The listing succeeded. That’s the failure mode worth building around: in a document portal, the metadata leak is the one that doesn’t announce itself.

The fix is a condition key on the list action, not a resource string:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOwnPrefixOnly",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::clinic-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "patients/${cognito-identity.amazonaws.com:sub}/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadOwnObjectsOnly",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::clinic-docs/patients/${cognito-identity.amazonaws.com:sub}/*"
    }
  ]
}

Two statements, two different jobs. The first constrains what the caller may ask the bucket to enumerate. The second constrains what bytes the caller may pull. Write only the second one and you’ve secured the download path while leaving the discovery path wide open.

One trap in that variable. ${cognito-identity.amazonaws.com:sub} resolves to the identity ID from the Cognito identity pool, not the sub claim from the user pool. They are different values for the same human. If you name your prefixes with the user pool sub because that’s what your application code has handy, the policy will match nothing and every request will be denied, which at least fails loudly.

Decide where the authorization boundary lives

There are two honest architectures here and the choice shapes everything downstream. Pick deliberately.

Pattern A: browser holds AWS credentials

The user signs in to a Cognito user pool. The application exchanges that token with a Cognito identity pool for temporary AWS credentials, and the browser calls S3 directly using the AWS SDK.

It’s less code and there’s no server to run. The authorization boundary is entirely the IAM role policy attached to the identity pool. That policy is the whole security model, so it had better be right, and it’s harder to reason about than application code because the enforcement happens in a service you can’t step through with a debugger.

The real limitation is expressiveness. IAM policy conditions do not know that Bob’s referral was superseded, that Alice revoked consent last Tuesday, or that a document should only be visible after a clinician releases it. Anything with business logic in it has to be encoded as prefix structure or object tags, and that gets brittle fast.

Pattern B: backend brokers every object

The browser never touches AWS credentials. It sends the Cognito access token to your own API. The API validates the token, checks its own records for who may see what, and returns a presigned URL for that one object. S3 is reachable only by the API’s execution role.

You now have a place to put logic, a place to log intent alongside access, and a single choke point to rate limit. The cost is that you own a service, its uptime, and its dependencies.

For a small clinic build, this is the one I reach for first. Not because Pattern A is insecure, but because the questions clients ask in month three (“can we hide results until the doctor reviews them?”) are answerable in Pattern B and require a re-architecture in Pattern A. The identity-pool approach earns its keep when access really is a pure function of who you are, which in healthcare it rarely stays.

If you do want attribute-driven access without a backend, Cognito identity pools can map claims from the token onto principal tags on the temporary session, which you then reference in policy conditions. It works, and it is the right tool for coarse distinctions like patient versus staff. It is not the right tool for per-document consent state.

Your download links expire before you think they do

This one generates support tickets on week two. You set a presigned URL to last a few hours. Patients report links dying in under one. Nothing in your code changed.

A presigned URL carries a signature made with whatever credentials signed it, and it cannot outlive those credentials. AWS documents the ceilings clearly: with long-lived IAM user keys and Signature Version 4 the maximum is seven days; from the S3 console the range is one minute to twelve hours; with temporary credentials from an assumed role, the URL dies when the role session does, regardless of the expiry you asked for. A default AssumeRole session is one hour.

Your Lambda or container signing those URLs is using temporary credentials. So the effective lifetime is the shorter of your requested expiry and the remaining time on that role session, and the remaining time depends on when the runtime last refreshed. That’s the source of the flapping. You can watch it yourself:

# Sign a URL and inspect what was actually encoded into it.
aws s3 presign s3://clinic-docs/patients/example/scan.pdf --expires-in 3600

# The returned URL contains X-Amz-Date, X-Amz-Expires and X-Amz-Credential.
# X-Amz-Expires is the window you asked for. If X-Amz-Credential shows an
# ASIA... key id, you signed with temporary credentials and the real ceiling
# is the session, not the window.

The honest resolution is to stop fighting it. Sign short, on demand, when the user clicks. Fifteen minutes is generous for a download the patient is standing in front of. Long links are not a convenience feature in this context anyway, because a presigned URL is a bearer credential: whoever holds the string can fetch the object, signed in or not. Forwarding the email forwards the access. If your portal mails links, you have moved the authorization boundary into someone’s inbox.

Mail a notification that says a document is ready and link to the portal login. Never mail the signed URL itself.

Encryption that actually gates something

New S3 objects are encrypted at rest by default with S3-managed keys. That satisfies the sentence “data is encrypted at rest” and gates precisely nothing, because anyone with s3:GetObject gets plaintext back. It is a storage property, not an access control.

Server-side encryption with an AWS KMS customer managed key is different in kind. Reading an object now requires two independent grants: the S3 permission and kms:Decrypt on the key. Those live in different policies and can be owned by different people. A bucket policy mistake no longer hands over plaintext on its own, and revoking the key grant cuts off reads without touching a single S3 permission.

You also get a second audit stream. Every decrypt is a KMS API call, recorded separately from the S3 call, which is useful precisely because an attacker who tampers with one trail probably didn’t think about the other.

The costs are real and worth stating. KMS charges per request, so a portal that fetches many small objects will notice. S3 Bucket Keys reduce that request volume substantially by caching a bucket-level key, at the price of coarser detail in the KMS audit records. For a document portal with modest traffic and high sensitivity, I’d take the per-request cost and keep the granularity. For a thumbnail-heavy application, the opposite.

The audit trail you don’t have yet

Assume someone will eventually ask: who downloaded this specific file, and when? If you have not explicitly turned something on, the answer is that you don’t know.

CloudTrail is on for management events by default, and management events are control-plane operations: creating the bucket, changing its policy, attaching a role. Object-level reads and writes are data events, and data events are off by default because of volume and cost. GetObject on a patient scan is a data event. Unless you configured a trail with data events for that bucket, no record of the download exists.

# Log S3 object-level activity for one bucket on an existing trail.
# Advanced event selectors let you scope to a single bucket prefix rather
# than every object in the account.
aws cloudtrail put-event-selectors 
  --trail-name clinic-audit-trail 
  --advanced-event-selectors '[
    {
      "Name": "PHI bucket object access",
      "FieldSelectors": [
        {"Field": "eventCategory", "Equals": ["Data"]},
        {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
        {"Field": "resources.ARN", "StartsWith":
          ["arn:aws:s3:::clinic-docs/patients/"]}
      ]
    }
  ]'

S3 server access logs are the other option and they are not equivalent. They record HTTP-level request detail and are delivered on a best-effort basis with no delivery guarantee, which makes them useful for traffic analysis and weak as evidence. CloudTrail data events carry caller identity, which is the field you actually need when the question is “who.”

Two things to sort out while you’re in there. Send the trail to a bucket in a different account, or at minimum a bucket the portal’s roles cannot write to, so that compromising the application doesn’t compromise its own log. And put a lifecycle rule on the log bucket, because data event volume grows with usage and nobody budgets for it. If you already run centralized dashboards on something like Grafana Cloud or CloudWatch, wire an alert to unusual read volume per identity now rather than after the incident.

Cutting off access when someone leaves

Disabling a Cognito user does not immediately end their session, and this surprises people. Tokens are self-contained. An access token already in a browser stays cryptographically valid until it expires, typically about an hour.

Refresh tokens are the longer exposure. The default is thirty days, and app clients can be configured anywhere from sixty minutes to ten years. Ten-year refresh tokens on a portal holding medical records is not a hypothetical misconfiguration; it’s a checkbox someone ticked to stop users complaining about re-login.

Cognito gives you revocation. Token revocation is enabled by default on newly created app clients. AdminUserGlobalSignOut revokes all of a user’s refresh, ID, and access tokens from the server side, and the /oauth2/revoke endpoint kills one refresh token and everything descended from it.

# Server-side, authorized with IAM credentials. Revokes the user's refresh,
# ID and access tokens.
aws cognito-idp admin-user-global-sign-out 
  --user-pool-id us-east-1_EXAMPLE 
  --username patient-example

# Confirm revocation is switched on for the app client before you rely on it.
aws cognito-idp describe-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --query 'UserPoolClient.EnableTokenRevocation'

Now the part that catches teams out. AWS documents that a revoked token still passes verification in a generic JWT library that only checks signature and expiry. Revocation is enforced by Cognito, not baked into the token. If your API validates tokens by fetching the JWKS and calling a verify function, a revoked token sails straight through until it expires on its own.

Which means short access token lifetimes are load-bearing. Revocation shortens the tail; it doesn’t close the window. Refresh token rotation, available on the Essentials and Plus feature tiers, replaces the refresh token on each exchange and narrows the window a stolen one is useful for. If you’re paying for Plus for threat protection features, you already have it.

Troubleshooting the usual suspects

Every request returns access denied after login

Nine times out of ten the prefix in your policy variable doesn’t match the prefix in your bucket. Print the identity ID your session actually resolved to and compare it character by character against a real key. Remember the identity pool ID and the user pool sub are different values.

Signature does not match on a presigned URL

Usually something modified the request after signing. A client library adding a header, a proxy rewriting the path, or an email gateway URL-encoding the query string. Compare the X-Amz-SignedHeaders list in the URL against what the client actually sent.

Uploads succeed but downloads fail

Check the KMS key policy separately from the IAM policy. A role can hold kms:GenerateDataKey (enough to write) without kms:Decrypt (needed to read). The S3 error will not say the word KMS prominently, which is why this one burns an afternoon.

Browser uploads fail before any request reaches S3

That’s CORS. The bucket needs a CORS configuration allowing your portal’s origin, the methods you use, and any headers the SDK sends. Browser preflight failures never appear in S3 logs, so check the browser console first.

Nothing appears in CloudTrail

Data events are not shown in CloudTrail Event History. Query the destination bucket directly, with Athena or CloudTrail Lake, and allow a few minutes for delivery.

Common mistakes

• Granting s3:ListBucket without an s3:prefix condition, so patients can enumerate every key even though downloads are blocked.
• Putting identifying information in object keys or file names. Assume the key will leak eventually and use opaque identifiers.
• Emailing presigned URLs. The link is the credential, and inboxes get forwarded, backed up, and breached.
• Extending refresh token lifetime to silence complaints about re-login, instead of implementing rotation.
• Assuming default encryption means access control. It means the disk is encrypted, nothing more.
• Enabling versioning without a lifecycle rule, so deleted documents linger indefinitely as noncurrent versions you forgot exist.
• Skipping CloudTrail data events to save money, then discovering the gap during an access review.

Best practices for a secure patient document portal

• Keep one bucket for patient documents and nothing else. Mixed-purpose buckets make policies ambiguous and audits slow.
• Turn on S3 Block Public Access at the account level, not just the bucket. Account level survives someone creating a new bucket in a hurry.
• Enable versioning and set a noncurrent version expiration rule, so an overwritten document is recoverable but not permanent.
• Use a KMS customer managed key so decrypt authority is separable from S3 authority.
• Require MFA for staff accounts. TOTP costs nothing to enable and staff accounts are the ones with breadth.
• Sign download URLs on demand with short expiry, and always from a backend that logged why it signed.
• Put the portal behind a WAF and rate limiting. Cloudflare in front of the application, or AWS WAF on the distribution, both work; the point is that credential stuffing against a login page is the most likely thing to actually happen to you.
• Restrict administrative access to the AWS console and to any bastion by network as well as identity, whether that’s IAM Identity Center with condition keys or a VPN such as NordVPN or Surfshark for a small team without central identity.
• Write down the retention period before launch, then encode it as a lifecycle rule. Retention decided later is retention never decided.

Frequently asked questions

Are S3 and Cognito enough for HIPAA compliance?

Both are covered under AWS’s HIPAA eligibility program, and Cognito’s documentation states it is HIPAA-BAA eligible. That makes them usable building blocks, not compliance. You need a signed Business Associate Agreement with AWS, and the configuration, access reviews, training, and documented policies remain yours. Service eligibility is a prerequisite, not an outcome.

Should I use a Cognito identity pool or a backend API?

Use an identity pool when access is a pure function of identity and the rules will not change. Use a backend when documents have states, when a clinician gates release, or when you need to log intent alongside access. Most clinic portals become the second within a year.

How long should a presigned URL last?

Minutes, not hours. Sign at click time. The maximum with long-lived credentials is seven days, but that ceiling is irrelevant here because the link is a bearer credential and a longer window is a longer leak.

Can patients upload documents back to the portal?

Yes, using presigned PUT or presigned POST. Presigned POST is worth knowing about because its policy document can constrain content type and size before the object lands, which presigned PUT cannot. Either way, write uploads into a quarantine prefix, scan them, then move them.

Does enabling CloudTrail data events cost much?

Data events are billed per event recorded with no free tier, unlike management events. For a portal serving a few thousand downloads a month the amount is small; for a bucket also serving application assets it is not. Scope your event selector to the patient prefix rather than the whole account and the question mostly goes away.

What happens to sessions when I disable a user?

Nothing immediate. Existing access tokens remain valid until they expire. Call AdminUserGlobalSignOut to revoke tokens, and keep access token lifetime short so the residual window is measured in minutes.

Do I need CloudFront in front of the bucket?

Not for correctness. It helps when patients are geographically spread, when you want signed cookies for a set of objects rather than one URL per object, or when you want a single edge to attach a WAF to. For a single-clinic portal, direct presigned URLs are simpler and one fewer thing to misconfigure.

The one thing to remember

A signed-in token proves who someone is. It does not decide what they may reach. In a secure patient document portal the boundary is the policy condition, the KMS grant, and the code that decides whether to sign a URL at all, and every one of those is something you have to write on purpose.

The demo passes either way. The difference only shows up when someone reads a key name they shouldn’t have seen, and nothing in your system considers that an error.

If you build one of these, spend an hour before launch on exactly two things: sign in as a test patient and try to list the bucket, then ask your logs who downloaded a specific file last week. If either answer is uncomfortable, you’ve found your next task.

Need a second pair of eyes on your portal?

This is the kind of build where the gap between working and safe is invisible from the outside. I help small teams close it.

• Reviewing S3 bucket policies and Cognito identity pool roles for prefix isolation gaps, including the list-versus-get split most policies miss.
• Designing the boundary: identity pool credentials in the browser versus a backend that brokers presigned URLs, with the trade-offs written down for your stakeholders.
• Setting up KMS customer managed keys, key policies, and grant separation so decrypt authority is not implied by storage authority.
• Turning on scoped CloudTrail data events, building the Athena tables to query them, and alerting on abnormal per-identity read volume.
• Hardening the Cognito side: MFA, token lifetimes, refresh token rotation, and a revocation path that actually ends sessions.
• Writing the whole thing as Terraform or OpenTofu so the next change is a reviewed pull request rather than a console click.

Send me your bucket policy and your identity pool role, with the account IDs redacted, and I’ll tell you what a signed-in patient can actually see.

The post Building a Secure Patient Document Portal on S3 and Cognito appeared first on John Nessime.

That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.