That conversation is usually the first sign that an organisation bought two Oracle products without deciding what each one was for. It is a common outcome, and it is not because the products are bad. It’s because they are sold as a suite, they overlap on purpose, and the sales conversation is about capability rather than authority.

This post is a working comparison of the four Oracle Construction and Engineering products people most often confuse: Primavera P6 vs Unifier vs Aconex, plus Textura. One profile each, what it genuinely owns, where it wins, where it doesn’t, and then the part nobody covers: how to decide which system holds the authoritative number when three of them offer a cost module.

The question that actually sorts them

Feature lists will not separate these products. All four manage documents. Three of them track cost. Two of them hold a schedule. If you compare capability columns you will conclude that you need all four, which is exactly the conclusion the suite is designed to produce.

The question that does separate them is: whose record is this, and who else needs to write to it?

Two of these products are single-organisation systems. You run them, you own the data, and outside parties get in only if you give them a seat and they accept your governance. One of them is deliberately multi-organisation, where every company on the project holds its own workspace and decides what it shares. And one is a transactional network that touches money leaving your bank account, which makes it a finance system wearing a construction badge.

Get that axis right and the rest of the decision mostly falls out.

Primavera P6: the schedule, and only the schedule

P6 is a critical path method scheduling engine with resource and cost loading on top. It exists to answer questions about time: what drives the finish date, what happens if this activity slips, what does the resource histogram look like if we accelerate.

Two things confuse people here. First, P6 comes in two shapes. P6 Professional is the desktop client that planners actually live in. P6 EPPM is the enterprise web layer with the shared database, portfolio views and web services. Most organisations run both against the same database, and most planners will fight you to keep the desktop client.

Second, Oracle Primavera Cloud is a separate product, not P6 in a browser. Oracle says so on its own product page, which is unusually direct for vendor copy. Primavera Cloud shares a lineage with P6 and is positioned as the cloud destination, but the data model is not identical and things planners rely on in P6 do not all have equivalents. If someone tells you a move to Primavera Cloud is a hosting change, they have not tried it.

Where P6 wins

• Contractual programme scheduling, where the schedule is a deliverable and gets audited
• Forensic delay work, because the XER and PMXML formats are what every expert and every third-party analysis tool expects
• Resource-loaded planning across a portfolio with a shared resource pool
• Anywhere the client’s contract names it, which is more places than you would expect

Where P6 doesn’t

• Cost control. P6 has cost fields and they are useful for cash flow shape, but a resource-loaded schedule is not a commitment ledger. It has no concept of a purchase order lifecycle, a variation approval or a payment certificate
• Correspondence and document control. There is nothing here for transmittals, RFIs or drawing registers
• Collaboration across companies. Subcontractors do not get P6 logins, and you would not want them writing to your programme if they did

Primavera Unifier: a business process engine that happens to know about projects

Unifier is the one people describe worst, usually as “the cost one”. That undersells it and also sets bad expectations.

Unifier is a configurable workflow platform for capital programmes. You define business processes: a change order, a funding request, an invoice, a payment application, a site instruction. Each one has a form, a workflow with approval steps and routing rules, and a defined effect on the cost sheet. Oracle ships preconfigured processes and a no-code designer called uDesigner for changing them or building your own. Alongside the business processes sit the managers, including the Cost Manager, Schedule Manager and Document Manager.

It’s sold as base products, principally Project Controls and Facilities and Asset Management, which matters because Unifier’s remit runs past handover into operating the asset.

The honest trade-off with Unifier is this: its configurability is the reason to buy it and the reason implementations run long. A platform that can model any approval workflow will not tell you what your approval workflow should be. If your organisation has not written its cost control process down, Unifier will faithfully automate the argument you are having instead of the process you wanted.

Where Unifier wins

• Owner-side capital programme control, where budget, funding, commitments, forecast and actuals need to roll up across many projects
• Anywhere approvals need an audit trail with defined routing, delegation and thresholds
• Public sector and utility programmes with fund tracking, where money has strings attached and the strings must be reportable
• Organisations that want one governed process applied to every project rather than per-project improvisation

Where Unifier doesn’t

• Supply chain collaboration. It’s your instance, your governance, your seats. External parties can be given access, but they are guests in your system and they behave like it
• Drawing and model control at scale. The Document Manager is real, but it is not a common data environment and it is not built for design coordination
• Fast starts. Configuration is a project in its own right, and treating it as an install is the most common way to burn a first year

Aconex: the project record, owned by nobody in particular

Aconex is a common data environment for correspondence, documents, workflows and models across every organisation on a project. Document control, transmittals, RFIs, submittals, drawing registers, model coordination, field inspections.

Its defining property is not a feature. It is the data ownership model. Every organisation on the project gets its own workspace and controls what it shares and when. That neutrality is the entire commercial argument, and it is the thing you cannot replicate by inviting subcontractors into your instance of something else.

This matters more than it sounds. A designer who believes the main contractor can read their internal drafts will keep their real work somewhere else, and your “single source of truth” quietly becomes a partial one. The reason Aconex accumulates a complete project record is that nobody is afraid of it. Oracle’s own materials lean on that neutrality hard, and in this case the marketing claim and the engineering consequence are the same thing.

On the model side, Aconex Model Coordination is design-neutral and built on open BIM standards including IFC and BCF, so it sits alongside the authoring tools rather than replacing them.

Where Aconex wins

• Multi-party projects where the correspondence record will be read by lawyers later
• Design-build and joint venture delivery, where no single party should hold everyone’s data
• Document and model volume, which Oracle’s datasheets describe as unbounded on data and participants
• Handover, because the audit trail and the linked document set are the deliverable

Where Aconex doesn’t

• Portfolio-level capital planning. Aconex is organised around projects, not around a programme of investment decisions
• Asset lifecycle after handover. The record ends where the project ends
• Deep enterprise cost governance. Connected Cost exists and is genuinely useful for project cost, but if your organisation needs funding, capital planning and multi-year rollup, that is Unifier’s shape

Textura: money leaving the building

Textura came into Oracle by acquisition and still feels different from the rest of the suite, because it is solving a different class of problem.

Textura Payment Management runs the construction payment cycle: initiating a draw, subcontractor pay applications, markups and approvals, owner billing, collecting lien waivers and compliance documents, enforcing compliance holds, payment authorisation and disbursement. It’s a network product. Your subcontractors transact in it, and the value comes from them being there.

The compliance side is the part that decides whether it’s worth buying. If your risk is a subcontractor’s insurance lapsing quietly and nobody noticing until a claim, an automated payment hold tied to a missing document is a control you cannot easily build with a spreadsheet and good intentions.

Where Textura wins

• General contractors with a wide subcontractor base and a monthly draw cycle that eats accounts payable alive
• Jurisdictions with real lien exposure, where waiver collection is a legal control rather than paperwork
• Compliance enforcement at the point of payment, which is the only moment anyone reliably pays attention
• Audit trails on who approved what, signed what and released what

Where Textura doesn’t

• It is not a cost control system. It handles the payment leg, not the budget, forecast or commitment ledger
• It does not replace your ERP. It integrates with it, and that integration is part of the implementation cost
• Adoption is a supply chain problem, not a software problem. If your subcontractors will not onboard, you own a slower process than the one you replaced

The overlap trap, and why the cost module is where it bites

Here’s the failure mode that is invisible at procurement and expensive eighteen months later.

Three of these products can hold cost. P6 has cost-loaded activities. Unifier has a full Cost Manager. Aconex has Connected Cost. Nobody sets out to run three cost systems, but it happens gradually: the schedule gets cost-loaded because the client asked for an S-curve, the finance team runs the commitment ledger in Unifier, and the project team starts logging variations in Aconex because that’s where the correspondence already is.

Now you have three forecasts. They disagree. Not because anyone is wrong, but because they are answering different questions with different cut-off dates and different accrual rules. And the reconciliation lands in a spreadsheet maintained by one person, which is the outcome every one of these products was bought to prevent.

The fix is a decision, not a tool. Before anything is configured, write down one sentence per data class: which system is authoritative, and what the others are allowed to hold.

• Schedule dates and logic: P6 is authoritative. Everything else receives dates, nothing else edits them
• Budget, commitments and forecast: pick one. If you own Unifier, it is Unifier. If you don’t, Connected Cost is a reasonable home, but say so out loud
• Correspondence and controlled documents: Aconex if it’s on the project, and if it is, do not run a parallel drawing register anywhere else
• Subcontractor payment and compliance status: Textura, with the resulting actuals flowing to whichever system owns cost
• Everything the authoritative system does not own: read-only, refreshed by integration, and clearly labelled in reports as a copy

That list takes an afternoon and saves a year. It’s also the single most common thing missing from Oracle Primavera implementations I get asked to look at.

Integration is the real cost, and it has a shape

Once you own two of these, you own an integration. Budget for it as a workstream, not a checkbox.

Oracle offers more than one path. Primavera Gateway is the established broker for P6 and Unifier, with built-in providers per application and a canonical intermediate format so mappings survive version changes. In the P6 to Unifier direction, Gateway maps the P6 Project ID to Unifier’s Project Number, and what flows depends on how the schedule is built: a duration-based schedule sends durations, a resource-loaded one sends durations and units. Oracle also has a lighter integration platform for exchanging data between Unifier, P6, Aconex and Primavera Cloud, so check which one your version and deployment model actually supports before designing anything.

Aconex Connected Cost takes a different route, talking to P6 over its web services API. Two constraints from that integration are worth knowing before you draw an architecture diagram:

• You can configure both a P6 and a Primavera Cloud integration for the organisation, but an individual project can only link to one of them at a time
• Self-hosted or third-party-hosted P6 servers have to be approved by Oracle and reachable over the internet before the integration can be established. Oracle-hosted P6 skips that approval step

That second one has teeth if you are running P6 EPPM on your own infrastructure. “Internet accessible” and “exposed” are not the same thing, and the gap between them is where your security review lives. If you’re standing up middleware or a reverse proxy in front of an on-premise P6 or Unifier deployment, that box needs to be treated as production: a dedicated host from a provider like Contabo or InterServer rather than a corner of an existing server, TLS terminated properly, access restricted to known networks, and remote administrative access over a business VPN such as NordLayer or Surfshark rather than a port forward someone opened for a consultant and forgot about.

Back up the database host properly too, and test a restore. On-premise P6 and Unifier estates are frequently the last thing anyone thinks about until they need it, and imaging tools from vendors like O&O Software earn their keep on exactly this kind of long-lived, rarely-touched server.

How I’d decide

Work through this in order and stop at the first answer that fits. Buying more than you need is far more common than buying too little.

1. Name the pain in one sentence. “We cannot defend our programme in a delay claim” points at P6. “We cannot tell the board what the capital programme will cost” points at Unifier. “We cannot find the approved revision” points at Aconex. “Subcontractor payments take six weeks and compliance is a filing cabinet” points at Textura
2. Count the organisations that must write to the system. One, and you are choosing between P6 and Unifier. Many, and you are choosing between Aconex and Textura
3. Ask whether the problem outlives the project. Assets, maintenance and multi-year capital planning are Unifier’s territory. Anything that ends at handover is not
4. Check what the contract already mandates. Plenty of clients name a scheduling tool or a project CDE in the contract documents. That decision is made, so spend your effort elsewhere
5. Ask the reseller how each product meters. Per named user, per project, per transaction or per value processed changes the architecture, not just the invoice. A per-user product pushes you towards fewer, more senior users, and that changes who does data entry
6. Only then decide on a second product, and write the system-of-record sentence for every data class the two of them share before signing anything

Arguments that don’t survive contact

• “Unifier can do document control, so we don’t need Aconex.” Technically true, practically false on multi-party projects. The obstacle is not features, it is that other companies will not put their real correspondence in your instance
• “Aconex has cost, so we don’t need Unifier.” Fine for project-level cost. It falls down when you need portfolio funding, capital planning and asset lifecycle
• “Primavera Cloud is just P6 in the browser.” Oracle itself states that Primavera Cloud is not P6. Treat any migration as a data model exercise with a gap analysis, not a lift and shift
• “We’ll integrate it later.” Later means after users have created data under two incompatible breakdown structures. Agree the project and cost breakdown codes on day one, even if the integration itself waits
• “The suite integrates out of the box.” Connectors exist and they are good. Mapping your codes, your workflow states and your calendars is still work, and it is work that needs someone who understands both sides

Practices worth adopting early

• Standardise the project ID before anything else. Gateway maps P6’s Project ID to Unifier’s Project Number, and a free-text naming convention will hurt on the day you connect them
• Keep the schedule in one place and let it flow outward. Bidirectional date editing across two systems is a support ticket generator
• Pilot Unifier configuration on one real process end to end before configuring twenty. uDesigner makes it easy to build the wrong thing quickly
• Onboard the supply chain before go-live on Aconex or Textura. Network products fail on adoption, not on uptime
• Export your own data on a schedule. XER and PMXML for schedules, and regular extracts from the cloud products, so you are not negotiating from zero at renewal
• Write the system-of-record statement into the implementation charter, where people will actually see it

FAQ

What is the difference between Primavera P6 and Unifier?

P6 is a critical path scheduling tool that answers questions about time and resources. Unifier is a configurable business process and cost platform that answers questions about money, approvals and governance across a capital programme. They overlap slightly because P6 can hold cost data and Unifier has a Schedule Manager, but neither is a substitute for the other.

Do I need Aconex if I already have Unifier?

If the project involves several independent organisations exchanging drawings and correspondence, usually yes. Unifier’s Document Manager works well for your own records, but external parties are guests in your instance. Aconex gives every organisation its own workspace and control over what it shares, which is what makes the project record complete rather than partial.

Is Oracle Primavera Cloud replacing P6?

Oracle positions Primavera Cloud as the cloud platform and publishes migration guidance from P6, while continuing to sell and support P6 EPPM. Oracle also states plainly that Primavera Cloud is not P6. Plan on the two coexisting for now, and treat any migration as a data model exercise with a proper gap analysis rather than a hosting change.

Can Textura replace our accounting system?

No. Textura handles the construction payment cycle: pay applications, approvals, lien waivers, compliance holds and disbursement. It integrates with ERP systems rather than replacing them, and that integration is a real line item in the implementation.

Which product should a general contractor start with?

It depends on which pain is loudest. If schedules are being disputed, P6. If the project record is scattered across email, Aconex. If the monthly draw cycle and subcontractor compliance are consuming the finance team, Textura. Contractors reach for Unifier less often than owners do, because Unifier’s strengths are programme-level funding and governance.

How hard is it to integrate P6 with Unifier?

The mechanism is well trodden through Primavera Gateway, which ships providers for both. The hard part is not the connector, it’s agreement: matching project identifiers, deciding which system owns dates, and reconciling breakdown structures that were designed independently. Expect the mapping work to take longer than the technical setup.

What data should I be exporting for portability?

For schedules, take regular XER and PMXML exports, since those formats are what independent analysis tools and delay experts read. For the cloud products, set up scheduled extracts of documents, transaction records and audit trails. The point is not distrust of the vendor, it’s that a project record you cannot read without an active subscription is a liability during a dispute.

The one thing worth remembering

The Primavera P6 vs Unifier vs Aconex comparison, with Textura alongside them, is not really a features question. Each of these products is good at the thing it was built for, and Oracle is not trying to trick anyone by giving three of them a cost module.

The damage comes from owning two of them and never deciding which one is authoritative. Sort the products by whose record they hold and how many organisations write to them, pick the smallest set that covers your loudest pain, and write down the system of record for every data class before a single workflow gets configured.

Do that and the monthly report matches the cost system. Skip it and you’ll meet the spreadsheet.

Need help untangling this?

Most of the work in this area is not clicking through screens, it’s the plumbing and the decisions around it. That part I can help with:

• Writing the system-of-record map for your data classes before you commit to a second Oracle product, so cost and dates have one owner each
• Designing and building integrations between P6, Unifier, Aconex, Textura and your ERP, including the field mapping and error handling nobody scopes
• Hardening on-premise P6 EPPM or Unifier deployments that need to be reachable for a cloud integration, with proper TLS, network restrictions and audited access
• Automating schedule and cost data extraction into a warehouse or BI layer, so reporting stops depending on someone manually exporting on a Friday
• Building repeatable backup, restore-testing and disaster recovery for self-hosted Primavera estates
• Second-opinion reviews on an implementation or integration design that already exists and is not behaving

If you’re stuck on something concrete, send it over: an integration mapping spec, a Gateway error, a network diagram for an on-premise P6 that needs to talk to a cloud service. Easier to give you something useful when I can see the actual thing.

The post Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need? appeared first on John Nessime.

That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that “went fine.”

This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.

The cuts that break production are rarely the ones you fear

The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.

The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.

So the rule I work by: sequence cost work by blast radius, not by savings size. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.

First, find out what you are actually paying for

The default Cost Explorer view groups by service, which is close to useless here. “EC2 – Other” is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.

Group by usage type instead

Usage type names the actual mechanism being billed. Strings like NatGateway-Bytes, NatGateway-Hours, PublicIPv4:InUseAddress and EBS:VolumeUsage.gp2 tell you what to go fix. This pulls last month’s usage types sorted by cost:

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30

That uses GNU date, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.

Billing data cannot tell you where your bytes went

This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.

VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.

Tier zero: savings production cannot notice

Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.

The orphan sweep

AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.

# Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text

Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.

NAT Gateway is usually the largest risk-free win

A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.

The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.

Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.

Log groups that never expire

CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.

# Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30

Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.

Incomplete multipart uploads

Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.

Tier one: savings that move a performance envelope

Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.

gp2 to gp3, and the throughput trap

This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.

The trap is throughput, not IOPS. gp3’s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is “the nightly job got slower” rather than an alert.

Check VolumeReadBytes and VolumeWriteBytes in CloudWatch first, and provision throughput explicitly where the volume needs it:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text

Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.

Rightsizing without guessing

AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.

aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text

My working rules:

• Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.
• Change one dimension at a time, family or size, so you know what caused any regression.
• Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.
• Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.
• Write down the metric you expect to move and the threshold at which you revert, before you make the change.

S3 storage classes and the small-object trap

Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.

Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.

Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.

Tier two: commitments, which break budgets rather than uptime

Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:

• Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.
• Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.
• Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.
• Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.
• Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.
• Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.

Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.

One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.

Troubleshooting: the bill went down and something broke

The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:

• Latency got worse but CPU is fine. Look at storage. VolumeQueueLength climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling BurstBalance is the tell, and it usually starts hours before anything is user-visible.
• Intermittent timeouts after rightsizing. Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.
• Something stopped reaching a service. If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint’s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.
• Objects are slow or missing from an application path. Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.

Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.

Common mistakes

• Starting with the biggest number. Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.
• Committing before rightsizing. This locks in the exact waste you were trying to remove, for years.
• Accepting gp3 defaults on high-throughput volumes. Cheaper storage, halved throughput ceiling, no alert.
• Enabling Intelligent-Tiering on buckets full of tiny objects. Monitoring fees on objects that will never move a tier.
• Turning off monitoring to save on monitoring. Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.
• Doing it once. Cost drifts back within two quarters unless something structural stops it.

Best practices that make the savings stick

• Make cost visible before it is a crisis. AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.
• Tag at creation, in the IaC. Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.
• Put non-production on a schedule. Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.
• Make the cheap thing the default in the module. gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.
• Review commitment coverage on a schedule. Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.

Frequently asked questions

What is the fastest way to cut AWS costs without breaking production?

Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.

Why is my “EC2 – Other” line item so large?

It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.

Is migrating from gp2 to gp3 always cheaper?

No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.

Can I cancel a Savings Plan if my usage drops?

Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.

Should I use Spot Instances to cut costs?

For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.

How do I find out where my NAT Gateway traffic is going?

Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.

The one thing worth remembering

If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.

Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say “not yet” when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.

Need a second pair of eyes on your AWS bill?

This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:

• A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.
• NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.
• EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.
• CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.
• Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.
• Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.

If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.

The post Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook appeared first on John Nessime.

The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.

That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.

This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn’t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.

Why the exit path matters more than the launch

All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.

Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.

So the question is not “which is cheapest for a small site.” It is “how much of this stack is portable, and who operates it after handover.”

Amazon Lightsail: where it wins and where it doesn’t

Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.

Where it wins

• Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.
• The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.
• Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.
• IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.

Where it doesn’t

• Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the default VPC in the same account and the same region. Not a custom VPC. Not another account.
• Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.
• The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.
• No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.

Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail’s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.

Amazon EC2: where it wins and where it doesn’t

EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.

Where it wins

• Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.
• Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.
• Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.
• Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.

Where it doesn’t

• Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.
• Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.
• Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.
• You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.

The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.

Amazon ECS: where it wins and where it doesn’t

ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.

• ECS on Fargate bills per vCPU-hour and GB-hour of the resources you declare in the task definition, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.
• ECS on EC2 means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.

Where it wins

• The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.
• Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.
• Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.
• Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.
• If the app is already containerized and CI already builds images, you are most of the way there.

Where it doesn’t

• The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.
• Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.
• Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.
• Handover is harder. A client’s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.

The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.

# Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"

Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.

Lightsail vs EC2 vs ECS on the lines that move the bill

Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.

• Egress. Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.
• NAT gateway. Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.
• Load balancing. Lightsail’s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.
• Public IPv4. Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.
• Log ingestion. CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.
• Idle resources. Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.

A decision procedure for Lightsail vs EC2 vs ECS

Run these in order. The first one that gives a clear answer is the answer.

1. Does this workload need to talk privately to other AWS resources? If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.
2. Who operates this in a year? If the answer is “the client’s office manager” or “nobody,” bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.
3. Is the app already containerized, with CI that builds images? If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.
4. Is the load steady or spiky? Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.
5. How much egress? Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail’s bundled allowance is a real argument.
6. What is the audit story? If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.

If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works before you need it. The commands are short:

# Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records

Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.

If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:

aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1

You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.

Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.

Arguments that don’t survive contact

“Lightsail is just EC2 with a nicer console”

Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.

“Start on Lightsail, migrate to EC2 when you outgrow it”

Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.

“Fargate is serverless, so it scales to zero and costs nothing when idle”

The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word “serverless” gives you.

“Containers are better practice, so ECS is the professional choice”

Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.

“Reserved Instances make EC2 cheaper, so always pick EC2”

Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client’s behalf is a conversation, not a default.

FAQ

Is Lightsail cheaper than EC2 for a small website?

Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.

Can a Lightsail instance connect to an RDS database?

Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the default VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.

Does everything migrate when I export Lightsail to EC2?

No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel & WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.

Is ECS overkill for a single small container?

Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.

Why is my Fargate bill higher than the vCPU-hour rate suggests?

Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.

Should I use Lightsail container service instead of ECS?

Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.

The one thing worth remembering

Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.

One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.

Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.

Need a second opinion before you commit?

Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:

• Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses
• Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency
• Planning and executing Lightsail to EC2 migrations, including the parts that do not export
• Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it
• Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy
• Writing the handover runbook so the next person can operate it without calling you

If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.

The post Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads appeared first on John Nessime.

So you open the AWS DMS console. The task is green. Full load completed, change data capture running, zero errors. Data validation passed on every table in scope.

Both of those things are true at the same time, and that is the part that catches people out. An Oracle to Aurora migration almost never fails with a stack trace and a rollback. It fails with a green dashboard and a handful of rows that are subtly, permanently wrong.

This post is not a walkthrough of the console wizard. AWS documents that well enough. It covers the four families of failure I would go looking for before signing off on an Oracle to Aurora migration: large objects that arrive truncated, change capture that records less than you assume, the semantic gaps between Oracle and PostgreSQL that compile cleanly and behave differently, and the database state that DMS simply does not move for you. Then how to validate properly, and what to do when a task stalls.

Pick the Aurora engine before you pick the migration tool

Amazon Aurora comes in a PostgreSQL-compatible edition and a MySQL-compatible edition, and the choice shapes everything downstream. Do not let it get decided by whoever on the team happens to like MySQL.

Aurora PostgreSQL is the realistic default coming from Oracle, and the reason is procedural code. PL/SQL and PL/pgSQL are close enough that automated conversion gets you a long way: packages, cursors, exceptions, custom types, and analytic functions all have recognisable counterparts. PostgreSQL also has the orafce extension, which reimplements a chunk of Oracle’s built-in functions and packages inside a compatibility schema, so a lot of converted code can keep calling things it already calls.

The honest case for Aurora MySQL: if the Oracle database is essentially a table store with the business logic living in the application tier, and your team already runs MySQL in production, you will move faster on the engine you can operate at 3am. Migration difficulty is real, but so is the cost of running a database nobody on the rota understands. If your schema has no packages, no triggers worth keeping and no PL/SQL, that argument holds.

Where it falls apart is the moment you find twelve thousand lines of PL/SQL that someone will have to rewrite by hand. At that point the engine choice has stopped being an operational preference and become a rewrite project.

Schema conversion: SCT or the DMS console

There are two ways to convert the schema. The AWS Schema Conversion Tool is a desktop Java application that connects to both databases, reads the source objects, and emits target DDL plus an assessment report of everything it could not convert. DMS Schema Conversion is the same idea run from the DMS console using instance profiles and data providers, with no local install and no JDBC drivers to manage.

Start with the console version. It is less setup and the assessment report is the thing you actually want early: a list of objects with action items, so you can size the manual work before committing to a date. Fall back to the desktop tool when you need something the console flow does not cover, such as converting application SQL files or SQL*Plus scripts.

Either way, treat the assessment report as a project plan, not a progress bar. The red items are the schedule.

Failure family one: LOB columns that arrive shorter than they left

This is the invoice description problem, and it is the single most common way an Oracle to Aurora migration loses data without telling anyone loudly.

DMS gives you three ways to handle CLOB, NCLOB and BLOB columns:

• Full LOB mode moves every LOB regardless of size. DMS has no idea how big they are, so it moves them one at a time, piece by piece. Correct, and slow.
• Limited LOB mode asks you for a maximum size up front. DMS pre-allocates memory and loads LOBs in bulk, which is dramatically faster. Anything larger than the limit is truncated to the limit, and a warning goes into the task log.
• Inline LOB mode sets a threshold. LOBs under it go inline with the row; larger ones fall back to the full LOB path with a lookup against the source. Good when most of your LOBs are small and a few are not.

Read the middle one again. Limited LOB mode is the default answer to “the full load is too slow”, it is the mode most tutorials use, and it silently clips your data. The warning lands in the task log, mixed in with everything else, and nobody is reading task logs line by line on a table with four million rows.

Worse, the task still reports success. Truncation is not an error condition in DMS. It is expected behaviour for the mode you asked for.

Measure before you set the limit

The fix is boring: go and find out how big your LOBs actually are, per column, on the source. Run this against Oracle for every LOB column in scope.

-- Largest LOB values in a single column, biggest first.
-- dbms_lob.getlength returns length in characters for CLOB/NCLOB,
-- bytes for BLOB. Check the top rows, not the average.
SELECT dbms_lob.getlength(description) AS lob_length
FROM   app.invoices
ORDER  BY dbms_lob.getlength(description) DESC
FETCH  FIRST 10 ROWS ONLY;

-- Same thing expressed in KB, which is the unit the DMS
-- "Max LOB size (K)" setting expects.
SELECT MAX(dbms_lob.getlength(description)) / 1024 AS max_kb
FROM   app.invoices;

Two things to know once you have the numbers.

First, with Oracle as a source, DMS treats LOBs as VARCHAR data wherever it can, because bulk-fetching them is much faster than the LOB API. Oracle’s VARCHAR ceiling is 32K, so a limited LOB size under 32K is where the mode performs best. If your real maximum is comfortably under that, limited LOB mode is genuinely the right call.

Second, memory. During full load, DMS pre-allocates roughly the max LOB size multiplied by the commit rate multiplied by the number of LOB columns. Push the limit up and that product grows fast. When the replication instance cannot allocate it, it starts swapping, and your “faster” mode becomes slower than full LOB mode. If limited LOB mode is dragging, drop the commit rate before you drop the size limit.

Split the LOB tables into their own task

The pattern I reach for first: one task with limited LOB mode for the bulk of the schema, a second task for the handful of tables with genuinely large LOBs, using full or inline mode. You can also override task-level LOB settings per table inside the table mapping rules, which keeps it to a single task if you prefer.

{
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 16,
    "InlineLobMaxSize": 0,
    "LobChunkSize": 64,
    "BatchApplyEnabled": false
  }
}

That is limited mode with a 16 KB ceiling. To use inline mode instead, set FullLobMode to true and give InlineLobMaxSize a non-zero value, which is the threshold below which LOBs travel inline.

One requirement that bites during ongoing replication: for CDC, tables with LOB columns need a primary key, because DMS looks the LOB value up in the source rather than reading it out of the redo stream. Full load does not care. CDC does. A LOB table with no primary key will load fine and then fail to replicate changes.

Failure family two: CDC that captures less than you think

Ongoing replication is what buys you a short cutover window. It is also where the assumptions hide.

DMS reads Oracle’s redo logs. For that to work at all, the source has to be in ARCHIVELOG mode with enough retention that DMS can still find the logs it needs after a pause, a network blip or an overnight stall. Retention is the one people get wrong: a task that stops for six hours against a two-hour retention window does not resume, it fails, and you restart from a full load.

Supplemental logging is not optional

Oracle’s redo logs, by default, contain enough information for Oracle to recover the database. That is not the same as enough information to reconstruct a row-level change for a different database. Supplemental logging is what closes that gap.

-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Identification key logging: writes every column of a row's
-- primary key into redo on UPDATE, even when the key didn't change.
-- Without this, DMS cannot reliably match the row on the target.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

-- Per-table alternative, if you only need a subset of tables
-- and want to keep the source overhead down.
ALTER TABLE app.invoices ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

The case that actually bites is a table with no primary key and no unique index. For those, key-level logging gives DMS nothing to work with, and it needs supplemental logging on all columns so the before-and-after image of the row lands in redo. Miss that, and DMS cannot build a safe UPDATE or DELETE statement for the target. You will not notice during full load. You will notice when an update on the source produces no change on the target, or changes the wrong row.

DMS ships premigration assessments that check exactly this. Run them. There are individual checks for supplemental logging on keyed tables, supplemental logging on all columns for unkeyed tables, ARCHIVELOG configuration, the CDC method in use, target permissions, and triggers enabled on the target. It takes minutes and it replaces a category of guesswork.

LogMiner or Binary Reader

DMS can read redo two ways. Oracle LogMiner is an Oracle-side API; DMS calls it and Oracle does the parsing. Binary Reader is DMS’s own parser reading the redo files directly.

If you specify nothing, you get LogMiner. That is usually correct. It needs fewer privileges, it handles ASM without extra plumbing, and it supports Oracle features like encryption and compression that the direct parser does not cover. The grants are small:

GRANT EXECUTE ON DBMS_LOGMNR TO dms_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO dms_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO dms_user;
GRANT LOGMINING TO dms_user;

Binary Reader earns its keep in one situation: a source generating redo faster than LogMiner can chew through it, or several concurrent migration tasks against the same database. LogMiner runs inside the source instance, so its CPU cost lands on the database you are trying not to disturb. Binary Reader shifts that work to the replication instance.

The trade-off is setup. Binary Reader needs additional privileges and file-level access to the redo logs, which means either a directory the replication instance can reach or a copy step. If your source is a busy production Oracle instance and CDC latency is climbing steadily rather than spiking, that is the signal to switch. Otherwise stay on LogMiner and spend the complexity budget elsewhere.

Whichever you pick, put CDC latency on a dashboard for the whole replication window, not just cutover day. The DMS task metrics publish to CloudWatch, and pulling them into whatever you already run for alerting, whether that is Grafana Cloud, Datadog or New Relic, is worth the twenty minutes. Source latency climbing while target latency stays flat means you are not reading redo fast enough. Both climbing together usually means the target cannot apply fast enough.

Failure family three: the semantics Oracle and PostgreSQL do not share

Schema conversion produces code that compiles. Compiling is not the same as behaving identically, and this is where a migration goes wrong in ways no tool flags.

Empty string is not NULL

Oracle treats the empty string and NULL as the same thing. PostgreSQL does not. This one difference propagates into comparisons, concatenation, unique constraints, and any procedural code that checks whether a value is “empty”.

-- PostgreSQL
SELECT '' IS NULL;        -- false
SELECT NULL || 'abc';     -- NULL
SELECT '' || 'abc';       -- abc

-- Oracle
SELECT CASE WHEN '' IS NULL THEN 'yes' ELSE 'no' END FROM dual;  -- yes
SELECT NULL || 'abc' FROM dual;                                  -- abc
SELECT '' || 'abc' FROM dual;                                    -- abc

Every one of those lines is a place a report can quietly change its numbers. Unique constraints are worse. Oracle stores an empty string as NULL, and NULLs are not compared for uniqueness, so it will happily accept many rows with an empty value in a uniquely constrained column. PostgreSQL sees zero-length strings, which are equal to each other, so the second one is rejected. Same schema, same data, different outcome, and the failure surfaces as an application error nobody can reproduce locally.

There is no switch for this. Either the application changes, or you wrap the affected functions, or you normalise on write with a trigger or a check constraint. Decide which, per column, before cutover. The orafce extension helps with function behaviour but it is not a global compatibility mode, and enabling the parts that change NULL semantics has its own consequences.

The rest of the list

• ROWID has no PostgreSQL equivalent. DMS Schema Conversion can emulate it with a bigint or a character varying column, but if application code depends on ROWID ordering or reuse semantics, emulation will not save you.
• SYSDATE and time zones. Conversion settings let you choose whether to emulate Oracle’s time zone handling or use native PostgreSQL behaviour. Native is faster. It is only safe if the database and the application genuinely run in the same time zone.
• TO_CHAR, TO_DATE, TO_NUMBER format masks. Oracle accepts parameters PostgreSQL does not. The converter emulates them by default, which is correct and slower. Turning emulation off is a per-codebase decision, not a global one.
• DUAL does not exist. PostgreSQL does not need a FROM clause, so most references just get dropped, but any dynamic SQL that builds a query string around DUAL needs finding.
• Synonyms have no direct equivalent. Search path adjustments and views cover most cases.
• Materialized views can be converted to real materialized views or to plain tables. The refresh semantics differ, so check what the reporting layer expects.
• NUMBER without a scale. An Oracle NUMBER column with no scale specified converts to something with a default, and if the source holds values wider than that default you get rounding or overflow. There is a premigration assessment specifically for this.

One data-level quirk worth knowing: if an Oracle column contains a NULL character (hex U+0000), DMS converts it to a space (U+0020) on a PostgreSQL target, because PostgreSQL cannot store it in a text column. Rare, but if you are migrating anything that stuffed binary into a VARCHAR, that is a real change to your data.

Failure family four: the state DMS never moves

DMS moves rows. It does not move everything that makes a database work, and the gaps are not obvious until you point traffic at the new cluster.

Sequences

Schema conversion creates the sequences. DMS copies the rows. Nothing advances the sequence counters to match the data that just arrived, so your target sequences sit at their starting value while the table already contains eight million rows.

The first insert after cutover collides with an existing primary key. Then the second. Then every write for the next eight million attempts.

Fix this after you stop replication and before you open traffic, not earlier, or you will do it twice.

-- Advance a sequence to match the data already loaded.
-- Run once per sequence, after CDC has stopped.
SELECT setval('app.invoices_id_seq',
              (SELECT max(id) FROM app.invoices));

-- Confirm before you open the doors.
SELECT last_value FROM app.invoices_id_seq;

Triggers on the target

If your converted schema has triggers and they are enabled during the load, they fire on every row DMS inserts. Audit tables get populated with a migration’s worth of fake activity. Derived columns get recalculated against half-loaded data. Anything that writes to another table doubles your write volume.

Disable target triggers for the load, re-enable at cutover. There is a premigration assessment that checks for enabled triggers on target tables in task scope, which is a decent backstop against forgetting.

Secondary indexes and foreign keys

Every secondary index present during full load is an index being maintained row by row while you are trying to move data as fast as possible. Foreign keys are worse, because DMS does not guarantee parent-before-child load order across tables.

Standard sequence: drop or disable secondary indexes and foreign key constraints before full load, recreate them afterwards, then let CDC catch up. Keep primary keys, since CDC needs them. Script the recreation and check it into version control alongside everything else, because rebuilding forty indexes from memory at 2am is not a plan.

Validation you can actually trust

DMS data validation compares source and target row by row and reports mismatches. It supports Oracle and PostgreSQL-compatible endpoints in both directions, so an Oracle to Aurora PostgreSQL path is covered. Turn it on.

Then understand what it will not tell you.

• It needs a primary key or unique index on both sides. Unkeyed tables are not validated, and those are exactly the tables most likely to have gone wrong.
• Collation and sort order differ between Oracle and PostgreSQL. When they disagree, validation reports failures on records that are actually fine, and the noise trains people to ignore the report.
• It does not run when the migration uses custom filtering, or when you are consolidating several source databases into one target.
• It costs real resources. Validation issues its own queries against both databases, on top of the migration traffic. Budget for that on the source.
• For LOB columns it compares checksums rather than values, using DBMS_CRYPTO on the Oracle side.

Because validation runs alongside migration, a task that stops takes validation with it. Validation-only tasks solve that: same endpoints, same table mappings, no data movement, running independently of the migration task. Use one when the validation load is hurting the source and you want to run it off-peak, or when you want validation to survive a migration task restart.

None of that replaces your own reconciliation. Before cutover, run business-level checks on both sides: rows per table, sums of every monetary column, min and max of every date column, distinct counts on anything you join on. Those catch things row comparison does not, because they answer the question the business actually asks.

And specifically for the LOB problem, compare maximum lengths, not just row counts.

-- On Aurora PostgreSQL, after the load.
-- Compare this against dbms_lob.getlength on the Oracle source.
SELECT count(*)                 AS row_count,
       max(length(description)) AS max_len,
       count(*) FILTER (WHERE description IS NULL) AS null_count
FROM   app.invoices;

If the target maximum is suspiciously round, exactly 16384 or exactly 32768, you have found your truncation.

Troubleshooting an Oracle to Aurora migration that stalls

Full load runs, then one table sits at zero

Almost always LOBs. Full LOB mode moves large objects one at a time with a source lookup per value, and on a table with a million large CLOBs that is a very long time with no visible progress. Check the table statistics for that table, then check whether it has LOB columns. Move it to its own task with limited or inline mode and an appropriate size.

CDC latency climbs and never comes down

Look at which latency is climbing. Source latency means redo is arriving faster than DMS can read it, which is the LogMiner to Binary Reader conversation, or a redo volume problem on the source. Target latency means Aurora cannot apply fast enough, which points at indexes still present, triggers still enabled, or an undersized writer instance.

Task fails on restart with missing archived logs

Retention. The logs DMS needed were removed before it came back. Increase archived log retention on the source to comfortably exceed your worst realistic outage, then restart from a fresh full load for the affected tables. There is no recovering the gap.

Updates on the source do not appear on the target

Supplemental logging, nine times out of ten. Check whether the affected table has a primary key or unique index, and whether logging is configured at the right level for that answer. The premigration assessments cover both cases.

Errors about a unique index on the target

DMS does not support replication to a table with a unique index built on a coalesce expression. If schema conversion produced one, or someone added it by hand to work around the empty-string problem, replication to that table will not work. Restructure the constraint.

Common mistakes

• Accepting the default limited LOB size without measuring the source. This is the truncation bug, and it has a specific, preventable cause.
• Treating a green task and a passed validation as proof of correctness. Both are true when data has been truncated.
• Skipping the premigration assessments because the wizard let you continue without them.
• Forgetting to advance sequences, then discovering it under production write load.
• Leaving triggers enabled on the target during full load and polluting audit tables.
• Assuming schema conversion handled empty strings. It converted the syntax, not the semantics.
• Setting archived log retention to the length of a normal outage rather than a bad one.
• Doing the first end-to-end rehearsal on cutover night.

Best practices

1. Run the assessment report first and size the manual work from it. The red items are your timeline, and they are better known in week one than week nine.
2. Inventory LOB columns before configuring anything. Maximum size per column, in KB, written down.
3. Separate the LOB-heavy tables into their own task. Different tables want different modes, and one global setting will always be wrong for someone.
4. Enable supplemental logging deliberately, per table class. Key-level for keyed tables, all-columns for unkeyed ones.
5. Drop secondary indexes and foreign keys for the load. Script the recreation, version it, test it.
6. Rehearse the whole cutover at least twice, including the sequence advance and the trigger re-enable. A scratch environment for the dry run does not have to be expensive: a temporary Aurora cluster you tear down afterwards, or a cheap dedicated box from somewhere like InterServer or Hetzner running a trimmed source copy, both work.
7. Run your own reconciliation alongside DMS validation. Counts, sums, min and max dates, maximum LOB lengths.
8. Keep the Oracle source running and readable for a while after cutover. The bugs that surface at week three are the ones you can only diagnose by comparing against the original.

Frequently asked questions

Should I migrate Oracle to Aurora PostgreSQL or Aurora MySQL?

Aurora PostgreSQL for anything with meaningful PL/SQL, because PL/pgSQL is close enough that automated conversion does most of the work, and the orafce extension covers a chunk of the rest. Aurora MySQL is defensible when the schema is plain tables, the logic lives in the application, and your team already operates MySQL. The deciding question is how many lines of procedural code you would have to rewrite by hand.

Can AWS DMS migrate stored procedures and packages?

No. DMS moves data. Schema and code conversion is a separate job, handled by DMS Schema Conversion in the console or the AWS Schema Conversion Tool on the desktop. Both convert most objects automatically and produce an action-item list for the rest.

Why is my DMS task green when data is missing?

Because truncation in limited LOB mode is expected behaviour, not an error. DMS writes a warning to the task log and carries on. Task status reflects whether the task is running, not whether the data is faithful. Compare maximum LOB lengths between source and target to catch it.

Do I need downtime for an Oracle to Aurora migration?

Some, but it can be short. Full load plus CDC means the target stays current while you test, so the outage is limited to stopping writes, letting CDC drain, advancing sequences, re-enabling triggers, recreating indexes and repointing the application. Rehearse it and that usually fits a maintenance window rather than a weekend.

Does DMS migrate sequences?

It copies row data, not sequence state. Schema conversion creates the sequence objects; you advance them yourself after stopping replication, using setval against the current maximum key value. Skipping this produces primary key collisions on the first write after cutover.

LogMiner or Binary Reader for Oracle CDC?

LogMiner unless you have a reason. It is the default, needs fewer privileges, and handles ASM, encryption and compression without extra work. Move to Binary Reader when redo volume is high enough that LogMiner cannot keep up, when you are running several tasks against the same source, or when the CPU cost of parsing on the source database is unacceptable.

Will DMS data validation catch every problem?

No. It needs a primary key or unique index on both sides, it can report false failures where Oracle and PostgreSQL collations disagree, and it does not run with custom filtering or multi-source consolidation. Treat it as one layer and add business-level reconciliation on top.

The one thing worth remembering

An Oracle to Aurora migration does not usually announce its failures. The task goes green, validation passes, and the damage shows up weeks later as a truncated field, a duplicate key, a report whose totals moved slightly.

So build the migration around proving correctness rather than proving completion. Measure your LOBs before you set a limit. Configure supplemental logging for the tables that have no key, not just the ones that do. Assume the semantic gaps between Oracle and PostgreSQL are still there after conversion, because they are. Advance the sequences. Then reconcile at the level the business cares about, not just row by row.

Green means the pipe is open. It does not mean what came out the other end is what went in.

Need a second pair of eyes on your Oracle to Aurora migration?

Most of the work on these projects is not running the wizard. It is finding the handful of things that will be wrong afterwards. That is the part I help with:

• Reviewing DMS task settings and table mappings for LOB truncation risk, per column, before you run a full load you cannot easily repeat.
• Working through the schema conversion assessment report and turning the red items into an actual estimate rather than a wall of warnings.
• Setting up supplemental logging and CDC correctly for tables with no primary key, including the LogMiner versus Binary Reader call for your redo volume.
• Writing the cutover runbook: index recreation scripts, sequence advance, trigger re-enable, ordered, tested, and timed against a real rehearsal.
• Building the reconciliation layer that sits on top of DMS validation, so you can sign off on the data with something better than a green tick.
• Post-cutover triage when something on Aurora behaves differently from Oracle and nobody can work out which layer changed.

If you have a task settings JSON, an assessment report, or a validation failure log you cannot make sense of, send it over and I will tell you what I see in it.

The post Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover appeared first on John Nessime.

That is the moment the AWS Amplify vs Firebase decision actually gets made. Not on day one, when both platforms look like a schema file, a typed client and a deploy that just works. On day two hundred, when the read path you committed to without noticing starts charging rent.

This post is a comparison, but not a feature tour. Both vendors publish good feature tables and neither of them is lying. What they do not tell you is which parts of the platform you can walk away from cheaply and which parts you are stuck with. So this is organised around the four things that decide that: the data model, the billing shape, the auth layer, and the exit cost.

The failure mode that bites: your read path is decided on day one

Both platforms hand you a pleasant abstraction over a NoSQL store. Firebase gives you Firestore, a document database. Amplify’s data layer, by default, gives you AppSync sitting in front of DynamoDB. In both cases you write a schema, get a typed client, and start shipping.

Here is what neither schema file makes obvious: it is not a description of your data. It is a bet on the queries you will need. Change the bet later and you are not editing a schema, you are backfilling.

In Firestore, every query shape needs an index that supports it. Filter on two fields and sort on a third and you need a composite index. The console will helpfully offer to create it for you, which is why most teams never notice they are accumulating a per-query-shape index budget. There are no joins. If an order needs its customer’s name, either you denormalise the name into the order document or you do a second fetch per row.

In Amplify, relationships between models are resolved by AppSync, which usually means an extra round trip per relationship. Secondary indexes have to be declared in the schema so the underlying table can be built to serve them. Same constraint, different vocabulary.

The good news is that both vendors have finally admitted this is a problem. Firebase Data Connect puts a managed Cloud SQL for PostgreSQL instance behind a GraphQL schema and generates typed SDKs against it. On the AWS side, Amplify’s data layer can be pointed at a SQL database rather than DynamoDB, and the newer Blocks approach lets you compose Postgres and other capabilities into an existing Amplify backend. If your application has anything resembling reporting, joins or ad-hoc filtering in its future, start there rather than starting on documents and migrating under pressure.

AWS Amplify: where it wins and where it doesn’t

Amplify Gen 2 defines the whole backend in TypeScript. Auth, data, storage and functions all live as code in an amplify/ directory, get reviewed in pull requests, and compile down through CDK to CloudFormation.

// amplify/data/resource.ts
import { type ClientSchema, a, defineData } from '@aws-amplify/backend';

const schema = a.schema({
  Order: a
    .model({
      region: a.string().required(),
      status: a.enum(['PENDING', 'SHIPPED', 'CANCELLED']),
      total: a.float(),
    })
    .authorization((allow) => [allow.owner()]),
});

export type Schema = ClientSchema<typeof schema>;

export const data = defineData({ schema });

Two things in that snippet matter more than they look. The authorization call is not a comment about intent, it generates the actual resolver-level access rules, so an access change is a code change with a diff. And ClientSchema flows the backend types into the frontend, which means removing a field breaks the build rather than breaking production at 3am.

Development happens in a per-developer cloud sandbox. It watches the amplify/ folder and redeploys on save:

# Deploy and watch your own isolated backend stack
npx ampx sandbox

# Tear it down when you're finished - it is real infrastructure
npx ampx sandbox delete

Where Amplify wins

• You are already on AWS. If your organisation has accounts, IAM boundaries, a VPC design and a compliance story, Amplify slots into all of it. Firebase would mean a second cloud to audit.
• The escape hatch is real. Because the backend compiles to CDK, you can drop into raw constructs and attach any AWS service without leaving the framework. That ceiling is much higher than Firebase’s.
• Environments are branches. Git branch to deployed environment is the native model, so staging is not a thing you build, it is a thing you push.
• Type safety end to end. This genuinely catches a class of bug that Firebase projects tend to find in production.

Where Amplify doesn’t

• Gen 1 is on a clock. Amplify Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. This matters because a lot of the tutorials, Stack Overflow answers and agency codebases you will encounter are Gen 1, and Gen 1 to Gen 2 is a migration, not an upgrade. AWS ships tooling for it, but the guide describes a blue/green run with an irreversible refactor step. Budget for it properly.
• “Local” development is not local. The sandbox is a real CloudFormation stack in a real account. It is high fidelity, which is the point, but your inner loop now includes deploy time and your dev environment costs money.
• Errors surface as AWS errors. A mistake in a TypeScript schema can come back as a CloudFormation rollback referencing a resource you never named. Debugging Amplify means being comfortable reading AppSync, DynamoDB and CloudWatch.
• Smaller community. Fewer answers exist, and a meaningful share of the ones that do exist are for the previous generation.

Google Firebase: where it wins and where it doesn’t

Firebase’s advantage is not that it is simpler. It is that the whole loop, from writing a rule to seeing a client behave, is shorter. The emulator suite runs the pieces on your machine:

# Run auth, Firestore and functions locally, no cloud project touched
firebase emulators:start --only auth,firestore,functions

The thing you will spend the most time on is security rules, and it is worth understanding why they are shaped the way they are:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /orders/{orderId} {
      allow read: if request.auth != null
                  && request.auth.uid == resource.data.ownerUid;

      allow create: if request.auth != null
                    && request.auth.uid == request.resource.data.ownerUid;

      allow update, delete: if false;
    }
  }
}

The distinction people get wrong is resource.data versus request.resource.data. The first is the document as it exists in the database, so it protects the read path. The second is the document the client is trying to write, so it protects the write path. Use the wrong one in a create rule and you are checking ownership on a document that does not exist yet, which evaluates to nothing and quietly lets the write through. That is the kind of bug that ships.

Where Firebase wins

• Mobile. Crashlytics, Cloud Messaging, Remote Config, A/B testing and analytics are one SDK away and genuinely good. AWS has equivalents, but not assembled like this.
• Offline and realtime. Firestore’s offline persistence and listener model handle flaky connectivity without you designing for it. If your users are on trains or in warehouses, this is a serious argument.
• Emulators. Being able to run the backend on a laptop, in CI, with no cloud project, is a real productivity difference against Amplify’s cloud sandbox.
• Time to first screen. For a prototype or an MVP that needs to exist by Friday, Firebase usually wins on speed alone.

Where Firebase doesn’t

• Configuration drifts out of git. The console is convenient and it is also a second source of truth. Indexes get created from an error link, a provider gets enabled by hand, and six months later nobody can rebuild the project from the repository. Terraform and the Firebase CLI both help, but you have to decide to use them.
• Security rules are a language, not a checkbox. They are also evaluated on the server for every access, which has cost implications when rules perform document lookups.
• Features you assume are included are an upgrade. Multi-factor auth, SAML and OIDC, blocking functions, audit logging and multi-tenancy live behind the Identity Platform upgrade, which comes with a different pricing model. On the free Spark plan, upgrading also introduces a daily active user cap. Find this out during design, not during an enterprise sales cycle.
• The ceiling is lower. When you outgrow Firebase you are usually moving to Google Cloud proper, which is a different set of tools and a different mental model.

What actually generates the bill

Rates change, so treat published numbers as current-at-time-of-reading and go to the vendor pricing pages for the arithmetic. What is stable, and what you should actually reason about, is the shape of the meters.

Firestore charges per document, and the details are where teams get caught:

• A query that returns 200 documents is 200 reads, whether you use two fields from each one or forty.
• A realtime listener is charged a read each time a document in its result set is added or updated. Attach a listener to a busy collection and you have built a meter that runs while nobody is looking at the screen.
• Where your SDK exposes query offsets, skipped documents are still billed. Paginate with cursors, not offsets.
• Aggregations such as count() are billed per batch of index entries read, with a minimum of one document read. Cheap, but not free, and not zero when you run them per page load.
• Stored bytes include indexes and metadata, so index sprawl shows up twice: once in write amplification, once in storage.

Amplify has no single meter, because Amplify is not a service. The bill is assembled from AppSync (query and mutation operations, realtime update messages, and connection-minutes for open WebSockets), DynamoDB, Lambda invocations and duration, CloudWatch log ingestion and retention, Cognito monthly active users, and Amplify Hosting build minutes plus data served.

That difference is the practical one. Firebase gives you a bill you can read in the console in thirty seconds. AWS gives you eight line items that will not tell you which feature caused them unless you tagged resources properly on the way in. If you go the AWS route, set up cost allocation tags and a budget alarm before launch, not after the first invoice. Pushing CloudWatch metrics into Grafana alongside your application metrics is worth the afternoon it takes, because cost anomalies and traffic anomalies are usually the same incident viewed twice.

Auth is the component you cannot cheaply replace

Firebase Authentication and Amazon Cognito are both fine. Neither is the reason to pick a platform. But auth is the layer where switching hurts most, because you are not migrating data, you are migrating sessions and identities, and every user notices.

Both let you export users, including password hashes, so a migration is technically possible. What is hard is everything hanging off the user ID: rules that compare against it, resolver authorisation that assumes it, third-party integrations keyed on it, and support tooling that looks users up by it.

One habit removes most of that pain regardless of which platform you pick: store your own internal user record from the first commit, with your own primary key, and treat the provider’s UID as an external identifier on that record. It costs you one table and one lookup. It converts an auth migration from a rewrite into a mapping exercise.

Hosting is the layer nobody should agonise over

Amplify Hosting deploys from a git branch and supports server-rendered frameworks. Firebase now has two: the original Hosting for static sites, and App Hosting for modern server-rendered apps, which is generally available and runs on Cloud Run with Cloud CDN in front.

Both are good. Neither should influence your decision much, because hosting is the one layer with genuine competition and near-zero switching cost. A Next.js or Nuxt app moves to Vercel, Netlify, Cloudflare Pages or a container on a VPS from a provider like InterServer or DigitalOcean in an afternoon. Put a CDN such as Cloudflare in front and the origin becomes an implementation detail your users never see.

Spend your deliberation budget on the data layer instead. That is where it pays.

Exit cost, ranked

If you rank the layers by what it costs to leave, the ordering is the same on both platforms and it is the opposite of the ordering people use when choosing:

1. Hosting. Swap it in a day. Barely counts as lock-in.
2. Functions. Cloud Functions and Lambda handlers are mostly your own code with a different signature wrapped around it. A rewrite, but a bounded one.
3. Auth. A user migration project with a communications plan attached.
4. The data model. You do not migrate this. You rebuild the read path. Firestore’s managed export produces Firestore-format files in Cloud Storage, and DynamoDB exports to S3, and in both cases what lands is denormalised documents shaped by decisions you made two years ago, not a relational dataset you can point a reporting tool at.

There is a fifth item with no clean export at all: realtime. Firestore listeners and AppSync subscriptions have no portable equivalent, so if realtime is core to your product, whichever you choose is close to permanent. That is not automatically an argument against either. It is an argument for knowing you are making that commitment when you make it.

AWS Amplify vs Firebase: how I’d actually decide

Not a scorecard. A procedure, in order, stopping at the first clear answer:

1. Write down your three hardest read queries. Not the CRUD. The reporting screen, the search, the admin filter. If you cannot write them, you are not ready to choose a backend.
2. Do any of them need joins, aggregates or filters you cannot predict? If yes, start relational: Firebase Data Connect, Amplify over a SQL data source, or plain Postgres behind an API you own. Do not start on documents planning to fix it later.
3. Is this mobile-first with real offline requirements? Firebase, and it is not close.
4. Does your organisation already run on AWS? Existing accounts, IAM, VPCs, audit posture and a security team who will ask questions about a second cloud all point to Amplify, even if Firebase would be faster this quarter.
5. Who is maintaining this in two years? Amplify Gen 2 expects TypeScript fluency and some tolerance for CDK and CloudFormation. Firebase expects less. Match the platform to the team you will actually have.
6. Any data residency or regulatory constraints? Check region availability for every service you plan to use, not just the headline one, before you commit.
7. Still tied? It is a prototype. Pick the one your team can ship this week, and write down explicitly that the data model is the part you will have to revisit.

Mistakes I see repeatedly

• Choosing on SDK ergonomics. Both SDKs are pleasant. You are picking a data model and a billing model, and the SDK is the layer you would replace most easily.
• Building reporting on the transactional store. Both platforms punish this. Stream to a warehouse or a read replica and let the operational store do one job.
• Launching without a budget alarm. A missing limit(), a listener on a busy collection, or one post that gets traction can turn a hobby bill into a real one overnight. Set the alarm before you need it.
• Treating authorisation as a later task. On both platforms, access control is coupled to the data model. Retrofitting it usually means reshaping documents.
• Following Amplify Gen 1 material. The commands, the directives and the mental model are all different. Check which generation a tutorial targets before you follow it.
• Letting console clicks become infrastructure. Anything you enabled by hand is something you cannot rebuild. Get it into the CLI config or into Terraform.
• Assuming the free tier is the plan. It is a trial of the plan. Model your costs at ten times current traffic and see whether you still like the answer.

Frequently asked questions

Is AWS Amplify or Firebase cheaper?

It depends entirely on your access patterns, and anyone who answers this without seeing them is guessing. Firebase tends to be cheaper and far more predictable at small scale, partly because the free allowances are generous and partly because there is one meter to watch. AWS tends to win when you have steady, high-volume traffic and the engineering discipline to tune DynamoDB capacity and Lambda sizing. The bigger practical difference is legibility: a Firebase bill tells you what happened, an AWS bill tells you what was consumed.

Can I move from Firebase to AWS Amplify later?

Yes, but price it by layer rather than as one project. Hosting and functions move cheaply. Auth is a real migration with user impact. The data model is usually a rebuild rather than a move, because a Firestore export gives you denormalised documents, not a relational dataset. Teams that manage this well are the ones that kept business logic out of security rules and out of client code from the beginning.

Is Amplify Gen 1 still supported?

Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. It receives critical fixes and security patches, not new features. If you are starting something new, use Gen 2. If you have a Gen 1 application in production, schedule the migration deliberately using the official tooling and guide, and test thoroughly before the refactor step, which is difficult to reverse.

Do I need Firebase Data Connect, or is Firestore enough?

Firestore is enough when your reads are predictable, mostly key-based, and denormalise cleanly. The moment you need joins, aggregates across collections, or filters that product will keep changing, Data Connect’s managed Postgres is the better foundation. It is also a lot easier to adopt at the start than to migrate onto after a year of document design.

Which one is better for a mobile app?

Firebase, in most cases. The offline persistence model, the crash reporting, the push messaging and the remote configuration are all mature and designed to work together. Amplify has mobile libraries and they work, but you are assembling the surrounding pieces from separate AWS services rather than getting them as a set.

What about a self-hosted alternative?

Worth considering if lock-in is your main worry. Supabase and Appwrite both offer a comparable developer experience over Postgres, and either can run on your own infrastructure. Understand what you are taking on, though: backups, restore testing, upgrades, patching and monitoring all become yours. That is a real operational load, and it is the trade most teams are implicitly paying a managed vendor to avoid.

Can I use both together?

You can, and some teams do run Firebase for analytics and messaging alongside an AWS backend. Be honest about the cost: two identity models, two billing accounts, two audit trails and two on-call runbooks. Do it because one platform is clearly better at a specific job, not because it was easier than making a decision.

The one thing worth remembering

The AWS Amplify vs Firebase question is almost never settled by features, and it is never settled by which SDK feels nicer in a tutorial. Both platforms will get you to a working application quickly, and both will hold up in production for the workloads they were designed for.

What separates them, months later, is the read path you committed to on day one and the layer you cannot afford to leave. Write down your three hardest queries before you write any schema. Decide whether you need relational storage while it is still a decision rather than a migration. Keep your own user record from the first commit. Set a budget alarm before launch.

Do those four things and either platform is a reasonable choice. Skip them and neither one saves you.

Need a second opinion before you commit?

Backend platform decisions are cheap to review and expensive to reverse. Things I help with in this area:

• Reviewing a Firestore or DynamoDB data model against the queries you actually need, before the design hardens
• Cost modelling both platforms against your real traffic assumptions, with the meters broken out so you can see what drives the bill
• Amplify Gen 1 to Gen 2 migration planning, including the blue/green sequence and what to verify before the irreversible step
• Auditing Firestore security rules and Amplify authorisation rules for gaps between the read path and the write path
• Setting up cost allocation tags, budget alarms and CloudWatch or Grafana dashboards so a runaway query shows up in hours rather than on the invoice
• Exit planning: working out what a migration off your current platform would genuinely cost, layer by layer

If you want a concrete answer rather than a general one, send me the thing itself: your schema file, your security rules, a screenshot of last month’s billing breakdown, or the three queries you are worried about. That is usually enough to tell you something useful.

The post AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave appeared first on John Nessime.

That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.

This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.

How Amazon Redshift actually charges you

Three buckets, and they behave very differently.

• Compute. On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.
• Storage. Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.
• Everything else. Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.

In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.

The billing mechanic that catches SaaS teams out

Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:

• The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.
• Usage is recorded when a transaction completes, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.
• Cancel a query before it finishes and you still pay for the time it ran.
• Querying system tables is billed like any other query. Your monitoring loop is a workload.
• After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.

Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.

Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.

-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;

charged_seconds is the column to build cost reporting on. compute_seconds is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.

Lever one: connections that look idle and are not

This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.

So a pool that fires SELECT 1 every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.

Open transactions are the same problem wearing a different hat. A BEGIN without a matching COMMIT or ROLLBACK keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.

What I would check, in this order:

1. Disable the pool’s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.
2. Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.
3. Fix any code path that opens a transaction and returns early on error without ending it.
4. Set a session timeout per application role so a leaked connection cannot bill indefinitely.

-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;

Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.

Lever two: base capacity, max capacity and usage limits are three different things

These get conflated constantly, and two of them will not save you a cent on their own.

• Base capacity (base RPU). The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.
• Max capacity (MaxRPU). A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.
• Usage limits. An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.

Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.

There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.

On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.

Lever three: Redshift cost optimization starts with knowing which tenant is expensive

Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.

Start by labelling every statement your API issues on a tenant’s behalf.

-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date >= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;

The label lands in the query log and surfaces as query_label in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.

-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time > dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;

Three columns in that view earn their keep beyond the obvious ones. result_cache_hit tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between queue_time and execution_time tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And user_query_hash groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.

From there, apportion the day’s charged_seconds by each tenant’s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between “Redshift costs us a lot” and “eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.”

Lever four: the isolation model you picked is a cost decision

AWS’s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.

• Pool. All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.
• Bridge. Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS’s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.
• Silo. A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.

Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer’s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.

In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.

Lever five: scan less, refresh less

Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.

• Sort keys that match your real predicates. Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.
• Materialized views for the panels every tenant loads. Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.
• Let the result cache work. Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.
• Tune zero-ETL refresh intervals. The refresh interval on the target database is adjustable via ALTER DATABASE. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.
• Keep cold history out of managed storage. Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.

One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.

Provisioned or serverless: how I would decide

Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.

Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.

Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant’s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.

Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.

The tell is simple. Pull a week of charged_seconds bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.

Troubleshooting: the bill moved and nothing shipped

1. Get hourly billed seconds first. Aggregate charged_seconds by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.
2. Check for anything running or queued right now. A single stuck statement explains a lot of otherwise inexplicable spend.
3. Look for transactions that never ended. A deploy that changed error handling can leave transactions open without a single failed request in your logs.
4. Compare query counts against query cost. Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.
5. Group by user_query_hash and diff against last week. New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.
6. Only then look at storage. Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.

-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;

Common mistakes

• Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.
• Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.
• Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.
• Building cost dashboards on compute_seconds instead of charged_seconds, then wondering why the totals never reconcile with the invoice.
• Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.
• Setting a usage limit to “turn off user queries” on the first day, before anyone knows what a normal week looks like.
• Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.

Best practices worth the effort

• Label every tenant-originated query with query_group from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool’s init SQL.
• UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.
• Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.
• Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.
• Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.
• Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.
• Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.

FAQ

Does Redshift Serverless really charge me when nobody is using the product?

Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.

How do I calculate the cost of a single query?

You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take charged_seconds for a period and divide it by each labelled tenant’s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.

Should I lower base capacity to save money?

Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.

Is a warehouse per tenant a good idea?

Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.

Does concurrency scaling cost extra?

On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.

Will a usage limit take my product down?

It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.

Why does my cost report never match the AWS invoice?

Usually one of three things: using compute_seconds rather than charged_seconds, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.

The one thing worth remembering

Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.

So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.

Need help getting your Redshift bill under control?

I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:

• Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.
• Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.
• Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.
• Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.
• Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.
• Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.

If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.

The post Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

Twenty seconds later the underlying disk no longer exists in AWS. Not detached. Deleted. The reclaim policy on the StorageClass was Delete, which is the default on essentially every managed cluster, and deleting the claim cascaded straight through to the volume.

That is the sharp edge of Kubernetes persistent volumes, and it is a naming problem as much as a technical one. Three objects sit between your pod and a real disk, the relationships between them are not obvious from the names, and the defaults are tuned for ephemeral test clusters rather than for anything holding data you care about.

This is short and covers the parts that bite: reclaim policies, what access modes actually mean, why volumes pin pods to a zone, and the difference between a StatefulSet and a Deployment that matters in practice.

Three objects, one disk

The mental model that prevents most of this:

• StorageClass is the recipe. It says which driver provisions volumes and what the defaults are.
• PersistentVolume is the actual piece of storage in the cluster. Usually created for you rather than by you.
• PersistentVolumeClaim is the request a pod makes, and the thing that binds one-to-one with a PV.

The PVC is the object your team touches, and it is the one whose name most understates its power. Deleting a PVC can delete a disk. That behaviour comes from the StorageClass, so check what yours says before you need to know:

# Read the RECLAIMPOLICY column. On managed clusters it usually
# says Delete, and nothing anywhere warns you about that.
kubectl get storageclass

For anything holding real data, define your own class with Retain and use it deliberately:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain              # PV outlives the PVC
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3

The trade-off is real and worth stating: Retain means orphaned volumes accumulate and quietly cost money, and a released PV will not automatically rebind to a new claim. You are choosing a cleanup chore over a data loss risk. That is the right trade for a database and the wrong one for a scratch cache.

Existing volumes keep whatever policy they were created with, so changing the class does nothing retroactively. Patch them individually:

kubectl patch pv "$PV_NAME" 
  -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

ReadWriteOnce does not mean one pod

It means one node. Multiple pods scheduled onto the same node can mount the same ReadWriteOnce volume and write to it simultaneously. The Kubernetes documentation says this explicitly, and almost everyone reads the name and assumes otherwise.

For a database, two writers on one filesystem is corruption. If you need genuine exclusivity, there is an access mode for it, stable since Kubernetes 1.29 and available on CSI volumes:

spec:
  accessModes:
    - ReadWriteOncePod        # enforced by the scheduler

The same access mode causes a second, more visible problem. Put a ReadWriteOnce volume on a Deployment with the default rolling update strategy and you get a deadlock: the new pod cannot start because the volume is still attached to the old pod’s node, and the old pod will not terminate until the new one is ready. The rollout sits there until it times out.

spec:
  strategy:
    type: Recreate            # old pod goes first, then the new one

That buys you downtime on every deploy, which is the honest cost of a single-writer volume behind a Deployment. If you cannot accept the downtime, you need either shared storage that supports ReadWriteMany or an application that handles its own replication, which is usually a StatefulSet.

Volumes pin pods to a zone

Block storage from a cloud provider lives in one availability zone. A pod using it can only ever be scheduled onto a node in that zone. Nothing in your Deployment spec says so, and the symptom is a pod stuck in Pending after a node replacement or a scale-up that happened to land elsewhere.

kubectl describe pod tells you, in among the scheduler events, that no node matched the volume’s node affinity. It is easy to miss if you are scanning for something more dramatic.

The preventive setting is volumeBindingMode: WaitForFirstConsumer, which is in the StorageClass above. It delays provisioning until a pod is actually scheduled, so the volume gets created in a zone where the pod can run rather than the other way round. Set it on every class you create. The immediate-binding alternative provisions a volume somewhere convenient and then constrains your scheduler forever.

StatefulSet or Deployment

The practical difference is not ordering guarantees or DNS names, useful as those are. It is that a StatefulSet gives each replica its own volume, created from a template, with a stable name that survives rescheduling. A Deployment with a PVC gives every replica the same volume, which is only correct when you have one replica.

So: one instance with a disk can be a Deployment with Recreate. Anything that scales and keeps per-replica state should be a StatefulSet.

StatefulSet PVCs historically survived everything, including deleting the StatefulSet, which is safe and leaves litter. Since Kubernetes 1.32 you can state the policy explicitly, and both fields still default to Retain:

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain       # deleting the StatefulSet keeps the data
    whenScaled: Delete        # scaling down cleans up the extra replicas

That pairing is the sensible default for most workloads. Be careful with whenScaled: Delete on anything where you scale down temporarily and expect the data back when you scale up again.

Two one-way doors

Volumes grow, they do not shrink. Expansion is stable and works by editing the claim’s requested size, provided the StorageClass allows it and the driver supports it. There is no supported path back. Over-provisioning a PVC is a permanent decision until you migrate the data to a smaller one by hand.

A volume snapshot is not a database backup. Snapshotting a running database’s disk gives you a crash-consistent image: the equivalent of pulling the power cable. Most engines recover from that, and “most” is not a word you want in a recovery plan. Use the database’s own backup mechanism for the database, and treat volume snapshots as infrastructure convenience rather than your restore path. Whichever you choose, restore one before you need to.

Troubleshooting

Pod stuck in Pending. Read the events at the bottom of kubectl describe pod. Volume node affinity conflict means a zone mismatch. No storage class means the cluster has no default and the claim did not name one.

PVC stuck in Pending. kubectl describe pvc names the provisioner error. With WaitForFirstConsumer this is normal until a pod is scheduled, so check whether anything is actually trying to consume it before debugging the driver.

Rollout hangs forever on a stateful app. ReadWriteOnce plus rolling update. Switch to Recreate.

PVC will not delete. A finalizer is holding it because something still mounts it. Find the consuming pod rather than removing the finalizer by hand, which is how you end up with a PV bound to a claim that no longer exists.

Volume is full but the PVC says it has room. You expanded the PVC and the filesystem did not follow. Most CSI drivers resize the filesystem on the next pod restart; if yours has not, restart the pod and check again.

Common mistakes

• Running production data on the default StorageClass without checking its reclaim policy.
• Assuming ReadWriteOnce prevents two pods from writing.
• A ReadWriteOnce volume on a Deployment with rolling updates.
• Scaling a Deployment with a PVC past one replica.
• Leaving volumeBindingMode: Immediate, then debugging Pending pods for an hour.
• Treating volume snapshots as database backups.
• Over-sizing a PVC on the assumption you can shrink it later.
• Removing a stuck finalizer instead of finding what is holding the volume.
• Deleting a namespace without checking what claims live in it.

Best practices

• Your own StorageClass with Retain for anything you would miss, and the default for everything else.
• WaitForFirstConsumer on every class you define.
• allowVolumeExpansion: true from the start, because adding it later does not help existing volumes.
• ReadWriteOncePod where exactly one writer is a correctness requirement.
• StatefulSets for per-replica state, single-replica Deployments with Recreate for everything else.
• An explicit persistentVolumeClaimRetentionPolicy rather than relying on the default.
• Application-level backups for databases, tested by restoring one.
• Alerts on volume capacity, because a full disk is the most predictable outage there is.

FAQ

Does deleting a PVC delete my data?

If the reclaim policy is Delete, yes, and that is the default nearly everywhere. Run kubectl get storageclass and read the RECLAIMPOLICY column before you assume otherwise.

Can two pods share a ReadWriteOnce volume?

Yes, if they are on the same node. ReadWriteOnce restricts access to one node, not one pod. Use ReadWriteOncePod when you need a single writer enforced.

Why is my pod Pending with a bound PVC?

Almost always zone affinity: the volume is in one availability zone and no schedulable node is there. The events in kubectl describe pod say so. WaitForFirstConsumer prevents it happening again.

Should I run databases in Kubernetes at all?

You can, with a mature operator that handles failover, backups and upgrades. Whether you should depends on whether your team wants to operate a database platform or just use a database. A managed service is often the better answer and there is no shame in it.

Can I shrink a PersistentVolumeClaim?

No. Expansion is supported, reduction is not. The only route down is a new smaller volume and a data migration.

The one thing to remember

Kubernetes storage defaults assume your cluster is disposable. Reclaim policies delete disks, access modes permit more sharing than their names suggest, and volumes constrain scheduling in ways nothing surfaces until a pod will not start.

Go and run kubectl get storageclass on your production cluster now. If the reclaim policy on the class holding your database says Delete, you have found something worth fixing before lunch.

Need a hand with stateful workloads?

Storage is where most Kubernetes migrations get uncomfortable, and it is usually the last thing anyone designs. Work I take on:

• Auditing a cluster’s storage classes, reclaim policies and access modes, and telling you what a stray kubectl delete would actually destroy.
• Moving stateful applications onto StatefulSets with sane retention, expansion and scheduling behaviour.
• Backup and restore design for data in Kubernetes, including an actual restore test rather than a documented intention.
• Diagnosing Pending pods, stuck rollouts and volumes that will not detach.
• CSI driver setup and storage class design on EKS and self-managed clusters.
• Capacity alerting so a full volume is a notification instead of an incident.

Send me the output of kubectl get storageclass and kubectl get pv, and I will tell you what stands out.

The post It’s Only a Claim: Kubernetes Persistent Volumes and Stateful Workloads appeared first on John Nessime.

The application behind all of this is one Go binary and a Postgres database. It serves a few hundred requests a minute. Nobody has touched its code in a fortnight.

That is the moment worth noticing: when Kubernetes is overkill, the tell is not that anything is broken. Everything works. It is that the cluster has quietly become the thing you operate, and the product has become a tenant.

What it actually buys you

Worth being fair, because the case for Kubernetes is real and I reach for it regularly.

It gives you bin-packing across many workloads, so a fleet of services shares machines efficiently. It gives you a declarative API that a platform team can build on, so twelve product teams deploy without twelve bespoke pipelines. It gives you self-healing, rolling updates, horizontal autoscaling and namespace-level isolation as defaults rather than as things you script. And it gives you a portable vocabulary: an engineer who knows Kubernetes knows most of your infrastructure on day one.

Every one of those benefits scales with the number of services and the number of teams. None of them scale with how modern you want to feel.

The bill nobody itemises

The cost is not the control plane fee. It is the surface area you have agreed to keep current.

Kubernetes ships three minor releases a year and supports each for about fourteen months, and you can skip at most one minor version when upgrading. So there is a permanent, non-negotiable upgrade cadence: roughly every four months something needs planning, testing and a maintenance window, forever. Fall behind and you are doing multi-hop upgrades on an unsupported version, which is the worst combination available.

Underneath that sit the components nobody counts when they say “we’ll just run Kubernetes”: a CNI plugin, a CSI driver, an ingress controller, cert-manager, external-dns, a metrics pipeline, an autoscaler, RBAC, and whatever operators your databases need. Each has its own release cycle, its own breaking changes and its own opinions about the others. That is where the Tuesday goes.

Then there is debugging. A request that used to fail in one place now fails in one of several, and telling apart an application bug from a NetworkPolicy, a readiness probe, a DNS caching issue or a node under memory pressure is a genuine skill. On a good team that skill exists. On a three-person team it exists in one person’s head and goes on holiday with them.

Signals you don’t need it yet

• You have fewer than about five services, and most of them are one binary and a database.
• One team owns everything, so there is nothing to isolate from anyone.
• Traffic is predictable enough that you size for peak and stop thinking about it.
• Nobody on the team has run a cluster in production before.
• Your deploys are already fine. Nobody is complaining about them.
• You cannot name the specific problem Kubernetes is solving, only the general one.
• The cluster would run on two nodes, which means you have a scheduler for a decision with two possible answers.

Signals you probably do

• Multiple teams deploying independently and tripping over each other’s environments.
• Enough services that per-service deployment scripts have become their own maintenance problem.
• Genuinely variable load where autoscaling saves real money, not hypothetical money.
• A compliance or tenancy requirement that maps cleanly onto namespaces and network policy.
• You are already running an operator-based product, like a database platform, that expects Kubernetes.
• Somebody’s whole job is the platform, and it is not also somebody’s whole job to ship features.

That last one is the honest test. Kubernetes is not expensive to install. It is expensive to own, and ownership needs a name attached to it.

What I’d reach for instead

Roughly in order of how much you take on.

One server, systemd, and a reverse proxy. Underrated to the point of being contrarian, and completely adequate for a large number of real businesses. A VPS from Hetzner, InterServer or DigitalOcean, nginx or Caddy in front, systemd units for your services, automated backups off the box. Restart policies and health checks are built in; you are not going without them, you are just getting them from init instead of from a control loop.

Docker Compose on one box. The step up when you want containers without a scheduler. The entire deployment is a file you can read in one screen:

services:
  app:
    image: ghcr.io/acme/api:1.4.2      # pinned, deployed by changing this line
    restart: always
    ports: ["127.0.0.1:8080:8080"]     # proxy handles TLS and the public port
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16                 # pin the major, upgrade deliberately
    restart: always
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes:
  pgdata:

The honest limitation: one box is one box. Losing it is an outage, and a redeploy is a brief gap unless you put something in front. For plenty of internal tools and early products, that is an acceptable trade stated out loud rather than an oversight.

A platform that runs containers for you. Cloud Run, App Runner, Fly.io, Render, Railway. You hand over a container image and get scaling, TLS, rolling deploys and health checks without operating any of it. This is the option most teams should look at hardest, because it buys back the majority of what Kubernetes offers at almost none of the operational cost. You pay in per-unit pricing and in reduced control, and both are usually worth it below a certain scale.

ECS on Fargate, or Nomad. Real schedulers with a fraction of the concepts. If you need orchestration but not the ecosystem, these are legitimate destinations rather than waypoints on the road to Kubernetes.

And if you have decided Kubernetes genuinely is the answer, use a managed control plane, and consider a lightweight distribution such as k3s for small or edge deployments. Do not build the cluster yourself to save money. That is the most expensive saving in infrastructure.

Arguments that don’t survive contact

“We’ll need it eventually.” Possibly. Adopting it three years early costs three years of upgrade cycles to buy an option you may not exercise, and the migration is not much harder later than it is now.

“It’s the industry standard.” For orchestrating fleets, yes. Most applications are not fleets. Standard does not mean universally appropriate, and the same argument would put a load balancer in front of a single server.

“It avoids vendor lock-in.” Partly true and frequently oversold. You have swapped a cloud provider’s API for a distribution, a set of controllers, and a pile of YAML that encodes assumptions about all of them. Portability lives in your container images and your data, not in the scheduler.

“It’ll help us hire.” It helps you hire people who want to run Kubernetes. Whether that is the same as the people who will build your product is worth a moment’s thought.

You’re not painting yourself into a corner

This is the part that makes the decision easy to defer, and it is the most useful thing in this post.

The durable investments are containerising your application, externalising configuration into environment variables, keeping state out of the application and in a database or object storage, exposing a health endpoint, and logging to stdout. Do those and your app already satisfies most of what a Kubernetes manifest asks for. The scheduler underneath is genuinely swappable.

So the choice is not Kubernetes now or a painful rewrite later. It is Kubernetes now, or the same containers running somewhere simpler until the complexity is earned.

FAQ

How many services before Kubernetes makes sense?

There is no clean number, and anyone offering one is guessing. The better question is how many teams deploy independently. One team can coordinate without a platform. Four cannot, and that is when the abstraction starts paying for itself.

Is Docker Compose really production-ready?

On a single host, for a workload that tolerates brief restarts, yes. It runs plenty of real systems. What it does not give you is failover across machines, so be clear that losing the host means an outage and decide whether that is acceptable rather than discovering it.

Doesn’t managed Kubernetes remove the operational burden?

It removes the control plane, which is the part that was never your main problem. Ingress, storage, networking, upgrades and the add-on ecosystem all remain yours. Managed offerings genuinely help, and they narrow the gap rather than closing it.

What about autoscaling? I can’t get that elsewhere.

You can. Cloud Run, App Runner, Fly and ECS all autoscale, several of them to zero, with no cluster to maintain. Autoscaling is a common reason to reach for Kubernetes and one of the weakest, because it is the feature most thoroughly commoditised elsewhere.

How hard is it to migrate later?

Much easier than migrating off it. If your app is containerised, stateless, configured by environment and observable, moving to Kubernetes is mostly writing manifests. The hard parts of that migration are the ones you should be fixing anyway.

The one thing to remember

Kubernetes is not complicated by accident. It is complicated because it solves a genuinely complicated problem, and if you do not have that problem you are paying the complexity and collecting none of the benefit.

So make it a decision with a reason attached. Write down the specific thing you cannot do today, and check whether a cluster is the cheapest way to fix it. Sometimes it is. Often the honest answer is one server, a reverse proxy, and an afternoon spent on backups instead.

Trying to decide?

I get called in on both sides of this: teams drowning in a cluster they did not need, and teams who genuinely need one and are putting it off. Work I take on:

• A straight assessment of whether Kubernetes fits your team and workload, with the reasoning written down rather than an opinion delivered.
• Building the simpler thing properly: single-host or Compose deployments with TLS, health checks, deploys and backups that hold up in production.
• Moving off an over-provisioned cluster onto something proportionate, without downtime.
• Getting an application genuinely portable so the decision stays reversible: containerised, stateless, configured by environment, observable.
• Setting up managed Kubernetes properly when it is the right answer, including the upgrade process nobody plans for.
• Reviewing the cost, in money and in engineering hours, of what you currently run.

Tell me how many services you run and how many people deploy them, and I will tell you what I would build.

The post The Cluster Becomes the Product: When Kubernetes Is Overkill appeared first on John Nessime.

The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.

Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word “delay” once.

Semantic similarity has no concept of a deadline. That single gap is the difference between an AI construction claims platform that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.

This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.

What a claims platform actually has to answer

Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.

1. Chronology. What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.
2. Entitlement. Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.
3. Causation. Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.
4. Quantum. What did the disruption cost? This is cost and resource data, joined to the events above.

Treat all four as “ask the documents” and you will get fluent nonsense on two of them. The architecture below splits them deliberately.

Failure one: retrieval that finds the best match instead of the first one

Two corpora live in every dispute bundle and they look identical to an embedding model.

• Contemporaneous records. Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.
• Claim-era material. Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.

Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.

The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus .metadata.json. So letter-0421.pdf gets letter-0421.pdf.metadata.json. The naming convention is the only link between them; there is no separate registration step.

{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}

Look closely at doc_date. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express “on or before the notice deadline”, only apply to NUMBER. Store the date as "2024-03-14" and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.

With the date as a sortable integer, a query filter can express the contractual window directly.

{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}

That is the whole trick. You are no longer asking “what is the most relevant document about this delay”. You are asking “what did the parties actually write during the window in which the contract required them to write it”. Those are different questions and only one of them is worth anything in a dispute.

Where the date comes from matters more than the date

Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the Date: header on an email, the printed date on a letter, the period covered by a diary entry.

Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a date_source attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.

Failure two: treating the programme like a document

This one is worse, because the output looks right.

Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.

Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.

Schedule data goes into a structured store, and the model queries it rather than reasoning about it.

1. Parse each programme file into tables. PyP6Xer handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.
2. Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.
3. Stamp every row with the data date of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project’s own view of itself.
4. Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.
5. Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.

A rough shape of the query that makes float erosion visible:

SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;

Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not “the model said so”.

Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.

Failure three: an answer with no paper trail

In most RAG applications a citation is a nice touch. In dispute work it is the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.

Design for that from the ingestion layer, not the presentation layer.

• Keep page and position. Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.
• Route low confidence to humans. Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.
• Reject ungrounded answers. Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.
• Keep an immutable evidential copy. S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.

One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International’s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs “the contractor is entitled to 42 days” is not helping. A system that outputs “here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing” is doing real work.

Failure four: one index for privileged and non-privileged material

Dispute bundles contain legal advice, counsel’s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.

The tempting shortcut is a privileged: false metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.

Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.

Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.

Choosing the AWS building blocks, including what not to build on

A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.

• Amazon Kendra entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.
• Amazon Bedrock Agents moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.

For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.

Managed Knowledge Base

AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client’s SharePoint rather than a bucket you control.

Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.

Custom Knowledge Base

You bring your own vector store and control chunking, embedding model and index layout.

Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn’t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.

Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.

Where the vector storage bill actually comes from

Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.

That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.

The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.

A reference pipeline

1. Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.
2. S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.
3. Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.
4. Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.
5. Derive the document date and write the .metadata.json sidecar. Anything undated goes to the review queue.
6. Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.
7. Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.
8. Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.

Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.

Troubleshooting

Date filters return nothing, and no error

Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in YYYYMMDD form and re-sync the affected documents.

A document is in the bucket but never appears in results

Check the sidecar filename first. It must be the complete original filename with .metadata.json appended, extension included. report.pdf.metadata.json works; report.metadata.json is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.

Answers cite the right document but the wrong passage

Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.

Float figures do not match the client’s own analysis

Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.

Query costs jumped without more usage

An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.

Common mistakes

• Using the file’s storage timestamp as the document date. It records the migration, not the event.
• Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.
• Embedding programme exports because they happen to be text files.
• Treating a metadata filter as a privilege boundary instead of an optimisation.
• Letting the model state entitlement conclusions rather than assembling and citing evidence.
• Building on services that have moved to maintenance mode because the tutorial you followed predates the change.
• Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.
• One index for every matter, which is both a cost problem and a confidentiality problem.

Best practices

• Make the document date a first-class, numeric, filterable attribute, and record where it came from.
• Keep an immutable evidential copy separate from the working copy the pipeline mutates.
• Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.
• Build a retrieval evaluation set from real questions with known correct documents before you tune anything.
• Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.
• Isolate privileged material by index and by IAM role, with metadata as a second layer.
• Log every query with its filters, so you can reconstruct how any given answer was reached.
• Define infrastructure as code so a new matter is a deployment, not a project.

FAQ

Can an AI construction claims platform replace a delay expert?

No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform’s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.

Should I use Amazon Kendra for the search layer?

Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.

How do I stop the model inventing float and delay figures?

Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.

Which vector store should I choose for a claims archive?

Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.

How do I handle scanned and handwritten site records?

Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.

Where do documents come from if they are not already in S3?

Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.

The one thing worth remembering

An AI construction claims platform lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.

Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.

Need help building this on AWS?

I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:

• Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.
• Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.
• Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.
• Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.
• Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.
• Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.

If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.

The post Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny appeared first on John Nessime.