That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.

This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.

How Amazon Redshift actually charges you

Three buckets, and they behave very differently.

• Compute. On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.
• Storage. Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.
• Everything else. Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.

In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.

The billing mechanic that catches SaaS teams out

Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:

• The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.
• Usage is recorded when a transaction completes, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.
• Cancel a query before it finishes and you still pay for the time it ran.
• Querying system tables is billed like any other query. Your monitoring loop is a workload.
• After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.

Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.

Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.

-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;

charged_seconds is the column to build cost reporting on. compute_seconds is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.

Lever one: connections that look idle and are not

This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.

So a pool that fires SELECT 1 every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.

Open transactions are the same problem wearing a different hat. A BEGIN without a matching COMMIT or ROLLBACK keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.

What I would check, in this order:

1. Disable the pool’s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.
2. Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.
3. Fix any code path that opens a transaction and returns early on error without ending it.
4. Set a session timeout per application role so a leaked connection cannot bill indefinitely.

-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;

Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.

Lever two: base capacity, max capacity and usage limits are three different things

These get conflated constantly, and two of them will not save you a cent on their own.

• Base capacity (base RPU). The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.
• Max capacity (MaxRPU). A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.
• Usage limits. An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.

Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.

There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.

On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.

Lever three: Redshift cost optimization starts with knowing which tenant is expensive

Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.

Start by labelling every statement your API issues on a tenant’s behalf.

-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date >= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;

The label lands in the query log and surfaces as query_label in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.

-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time > dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;

Three columns in that view earn their keep beyond the obvious ones. result_cache_hit tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between queue_time and execution_time tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And user_query_hash groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.

From there, apportion the day’s charged_seconds by each tenant’s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between “Redshift costs us a lot” and “eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.”

Lever four: the isolation model you picked is a cost decision

AWS’s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.

• Pool. All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.
• Bridge. Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS’s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.
• Silo. A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.

Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer’s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.

In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.

Lever five: scan less, refresh less

Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.

• Sort keys that match your real predicates. Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.
• Materialized views for the panels every tenant loads. Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.
• Let the result cache work. Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.
• Tune zero-ETL refresh intervals. The refresh interval on the target database is adjustable via ALTER DATABASE. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.
• Keep cold history out of managed storage. Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.

One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.

Provisioned or serverless: how I would decide

Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.

Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.

Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant’s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.

Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.

The tell is simple. Pull a week of charged_seconds bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.

Troubleshooting: the bill moved and nothing shipped

1. Get hourly billed seconds first. Aggregate charged_seconds by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.
2. Check for anything running or queued right now. A single stuck statement explains a lot of otherwise inexplicable spend.
3. Look for transactions that never ended. A deploy that changed error handling can leave transactions open without a single failed request in your logs.
4. Compare query counts against query cost. Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.
5. Group by user_query_hash and diff against last week. New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.
6. Only then look at storage. Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.

-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;

Common mistakes

• Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.
• Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.
• Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.
• Building cost dashboards on compute_seconds instead of charged_seconds, then wondering why the totals never reconcile with the invoice.
• Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.
• Setting a usage limit to “turn off user queries” on the first day, before anyone knows what a normal week looks like.
• Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.

Best practices worth the effort

• Label every tenant-originated query with query_group from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool’s init SQL.
• UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.
• Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.
• Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.
• Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.
• Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.
• Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.

FAQ

Does Redshift Serverless really charge me when nobody is using the product?

Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.

How do I calculate the cost of a single query?

You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take charged_seconds for a period and divide it by each labelled tenant’s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.

Should I lower base capacity to save money?

Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.

Is a warehouse per tenant a good idea?

Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.

Does concurrency scaling cost extra?

On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.

Will a usage limit take my product down?

It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.

Why does my cost report never match the AWS invoice?

Usually one of three things: using compute_seconds rather than charged_seconds, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.

The one thing worth remembering

Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.

So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.

Need help getting your Redshift bill under control?

I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:

• Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.
• Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.
• Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.
• Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.
• Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.
• Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.

If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.

The post Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill appeared first on John Nessime.

Eighteen months later there are forty flows, several of them on five-minute schedules because someone wanted fresher dashboards, and AppFlow is a line item that gets mentioned in the monthly review. Nothing was done wrong. The spreadsheet modelled the situation at the moment of the decision and that situation changed.

This is what makes an honest AppFlow vs Lambda vs MuleSoft cost comparison hard: you are not comparing three prices, you are comparing three different shapes, and the cheapest one at two integrations is frequently not the cheapest one at forty. The crossover is the entire decision, and almost nobody models it.

A note on numbers before we start. I am not quoting rate cards. AWS pricing varies by region and changes, MuleSoft publishes no list prices at all, and every comparison article with a neat table of dollar figures is quietly out of date. What does not change is the mechanism each one bills on, and that is what determines whether your bill grows with volume, with integration count, or not at all.

Three shapes

• AppFlow is pure consumption with no floor. You pay per flow run and per gigabyte processed. Two flows cost almost nothing; the bill scales with flows multiplied by frequency.
• Lambda is consumption too, but the meter is nearly irrelevant. The real cost is engineering time to build and own it, and the surrounding AWS services it drags along.
• MuleSoft is a floor. An annual contract sized on capacity or usage entitlements, paid whether you use it or not, largely independent of how many integrations you run inside it.

Draw those as lines against integration count and you get the whole argument. One starts near zero and climbs steeply. One starts moderate and climbs gently. One starts high and is close to flat. Where they cross depends entirely on your situation, which is why a generic answer is worthless and a framework is not.

AppFlow: cheap until the schedule tightens

AppFlow bills per successful flow run plus data processed. The detail that catches people, and it is stated plainly in AWS’s own pricing documentation, is this: a flow run that checks for new data and finds none still counts. You pay for polling, not for movement.

That turns schedule frequency into the dominant variable, and it is the one nobody models because at decision time everything is daily:

Flow runs per month, per flow:

  every 5 minutes   12 x 24 x 30  =  8,640
  every 15 minutes   4 x 24 x 30  =  2,880
  hourly                 24 x 30  =    720
  daily                       30  =     30

Twenty flows, five-minute schedule:  172,800 runs
The same twenty flows, hourly:        14,400 runs

Identical data. Twelve times the flow run charge.

Two more things that are not on the AppFlow line item. Standard S3 request and storage charges apply to whatever it writes, and KMS charges apply to the keys it uses. And on some connectors, running additional concurrent processes to speed a transfer is billed as additional flows.

Where AppFlow wins: a small number of standard SaaS-to-AWS movements on relaxed schedules, where nobody wants to own code. It is genuinely excellent at that, it goes live in an afternoon, and there is no floor to justify.

Where it stops winning: many flows, tight schedules, or transformation logic it cannot express. Schema drift handling is basic, and there is a point where you are working around the tool more than using it.

Lambda: the invoice is not the cost

For CRM-sized workloads, Lambda’s compute charge is close to a rounding error. A function that runs hourly for thirty seconds costs so little that optimising it is a waste of an afternoon.

The costs are elsewhere, and two of them are large.

The surrounding AWS services. NAT gateway data processing if the function sits in a VPC, CloudWatch Logs ingestion, S3 requests from small-file writes, Athena scans downstream. On a real pipeline these routinely exceed the compute charge several times over, and none of them appear under “Lambda” in Cost Explorer.

Engineering time, forever. Not just the build. Pagination, retries, watermarks, delete handling, schema drift, credential rotation, alerting, and someone available when it breaks at an awkward hour. This is the line that gets costed at zero in every comparison and is usually the largest number in the exercise.

The interesting property, though, is that the marginal cost of integration number eleven is much lower than integration number one, if you built a framework rather than eleven scripts. Shared extraction, shared state, shared error handling, one deployment pipeline. That is what makes the Lambda line climb gently instead of steeply. Teams that build eleven bespoke functions get the worst of both worlds: consumption pricing with none of the amortisation.

Where Lambda wins: you already have AWS engineers, you need control over the output shape, and the integration count is going up. Also anywhere the requirement is genuinely unusual, because a managed connector cannot express what it does not have a checkbox for.

Where it does not: a team of two who would rather ship product, or an organisation where the integration needs to be maintainable by people who are not engineers.

MuleSoft: a floor, and what it buys

MuleSoft publishes no list prices. Everything is quote-based, annual, and negotiated, which means any figure you find online is somebody else’s contract and tells you little about yours. It is also worth knowing that the commercial model has been repackaged relatively recently, moving new customers away from the older capacity-based structure toward usage entitlements, so a pricing article from a couple of years ago may describe a product you cannot buy.

Three structural things matter more than the number.

It is a commitment, not a meter. You size it up front and pay for it regardless of use, which is bad if you are unsure and fine if you are certain. Undersizing means overages; oversizing means paying for headroom.

Separate meters do not offset each other. Integration capacity, API traffic and add-on modules are metered independently, so spare room in one does not cover a spike in another. Model each axis separately or the renewal will be a surprise.

Renewal escalation is normal. Enterprise contracts typically carry annual uplift clauses, so the three-year total is meaningfully above three times year one. Negotiate that at signature, because the leverage is gone afterwards.

Now the fair part, because a post that treats MuleSoft as merely expensive is not being honest. What the floor actually buys: API management and governance as a first-class capability rather than something you assemble; a connector catalogue covering legacy and on-premise systems that neither AWS option touches; integrations that a trained analyst can maintain without a software team; audit trails, support contracts and SLAs that satisfy people who ask for them in writing. At a hundred integrations across several business units, the fixed cost is spread thin and the governance is the point.

The cost that is easy to miss is skills. DataWeave and the Anypoint tooling are proprietary, so you are hiring for a specific platform and your switching cost rises with every integration you build. That is not an argument against it; it is an argument for deciding deliberately rather than drifting in.

Where MuleSoft wins: many integrations, many teams, on-premise or legacy systems in the mix, and governance requirements with a compliance function attached to them.

Where it does not: three integrations, one team, all cloud. The floor will never amortise and you will spend the first year justifying it.

Modelling the crossover

Build the comparison over three years rather than one month, and project the inputs rather than freezing them.

1. Count integrations at year three, not today. Ask the people requesting them. The answer is always higher than the current number and it is the variable the whole model turns on.
2. Write down the schedule for each one, and defend it. “Every five minutes” is usually a preference, not a requirement. For AppFlow this is the difference between a small bill and a large one.
3. For AppFlow, multiply flows by runs per month. Apply current rates yourself, then add S3 requests and KMS.
4. For Lambda, cost the engineering explicitly. Build days plus ongoing maintenance days per year, at a loaded rate. If that number embarrasses you, it is probably the accurate one.
5. For MuleSoft, get an actual quote, model each meter separately, and include renewal escalation across the term.
6. Add the cost of being wrong. How much does switching cost in each direction? Consumption services you can walk away from. An annual contract and a proprietary skill set you cannot.

Then plot all three. You are looking for where the lines cross and whether your projected integration count sits comfortably on one side of it or uncomfortably near it. Near a crossover, pick the option that is cheaper to reverse.

Common mistakes

• Comparing one month at today’s integration count.
• Costing engineering time at zero on the build-it-yourself option.
• Not modelling flow run frequency, which is the single largest AppFlow variable.
• Forgetting that AppFlow charges for polling runs that find nothing.
• Attributing only the Lambda invoice to the Lambda option, ignoring NAT, logs and S3 requests.
• Building eleven bespoke Lambdas instead of one framework, so nothing amortises.
• Using published MuleSoft prices from an article, when no list prices exist and the packaging has changed.
• Ignoring renewal escalation when comparing an annual contract against consumption pricing.
• Treating a hybrid as a failure of nerve rather than the usual right answer.
• Choosing for the architecture you want rather than the team you have.

How I’d decide

Start with AppFlow if the integrations are standard, few, and not urgent. It is the lowest-commitment option and you can leave whenever you like.

Move to Lambda when you have AWS engineers, the count is climbing, and you need control over the output. Build it once as a framework, not repeatedly as scripts, or you get the cost profile without the benefit.

Consider MuleSoft when the problem is organisational rather than technical: many teams, legacy systems, governance requirements, and integrations that need to outlive the person who built them.

And be relaxed about mixing them. Plenty of sensible estates run AppFlow for the boring standard movements and custom code for the two integrations that are genuinely specific. Insisting on one tool for everything is a preference, not an architecture.

FAQ

Which is cheapest overall?

None of them, universally. AppFlow is cheapest at low integration counts and relaxed schedules. Lambda is cheapest at scale if you already have the engineers and build a shared framework. MuleSoft is cheapest only when the fixed cost is spread across many integrations and teams. The question is which shape fits your trajectory.

Why did my AppFlow bill grow when data volume didn’t?

Because you are billed per flow run, including runs that check for new data and find none. Somebody tightened a schedule, or the flow count grew. Both multiply the run charge without moving a single extra record.

Is Lambda really cheaper than AppFlow?

On the invoice, usually yes. In total cost, only once the engineering amortises across several integrations. For one or two simple flows, AppFlow is almost always the better economic answer, and the “we’ll just write a Lambda” instinct is a false economy at that scale.

Why can’t I find MuleSoft’s prices?

Because there aren’t any published. It is sold through quotes with negotiated discounts, so figures you find online are other people’s contracts. Get a quote, model each meter separately, and negotiate the renewal escalation before you sign rather than after.

Can I start on one and move later?

Between the two consumption options, fairly easily, especially if your destination schema is defined by you rather than by the tool. Off an annual contract with proprietary tooling, much less easily, which is exactly why the switching cost belongs in the model rather than as a footnote.

The one thing to remember

You are not choosing the cheapest tool. You are choosing a cost shape, and committing to how it behaves as your integration count and your schedules change over the next three years.

So model the trajectory, not the snapshot. Put engineering time in the spreadsheet at a real rate. And when the lines are close together, take the option you can walk away from, because being wrong cheaply is worth more than being right narrowly.

Need the model built for your numbers?

This decision usually gets made under time pressure with incomplete inputs, and then lived with for years. Work I take on:

• Building the three-year cost model for your actual integration inventory, schedules and volumes, with engineering time costed honestly.
• Auditing an existing AppFlow estate for flow count, schedule frequency and the charges that sit outside the AppFlow line item.
• Designing a shared extraction framework so custom integrations amortise instead of multiplying.
• Migrating between approaches without downtime, including keeping the destination schema stable so the choice stays reversible.
• Reviewing an integration platform contract for meter structure, overage terms and renewal escalation before you sign.
• Hybrid architectures where managed connectors handle the standard movements and code handles the awkward ones.

Send me your integration list with schedules and rough volumes, and I will tell you which shape you are actually buying.

The post Cheapest at Two Flows, Worst at Forty: AppFlow vs Lambda vs MuleSoft on Cost appeared first on John Nessime.

“EC2 – Other” is where AWS files NAT gateway charges, along with EBS, elastic IPs and assorted data transfer. Your Salesforce sync runs in a private subnet, so every byte it pulls from Salesforce and every byte it writes to S3 goes through a NAT gateway, which bills an hourly rate plus a charge per gigabyte processed. None of that appears under Lambda, or under S3, or under the tag you put on the pipeline.

That is the recurring theme when you look at the AWS cost of Salesforce data pipelines: the spend is almost never in the place you have been optimising. People tune Lambda memory settings while the bill is being driven by data movement, request counts, log ingestion and scanned bytes.

Five levers, in the order I would pull them.

First, find out where it actually goes

Every hour spent guessing is an hour not spent on the line item that matters. Tag everything the pipeline owns with a consistent cost allocation tag, activate it in the billing console, and wait for it to backfill.

Then group Cost Explorer by usage type rather than by service. Service-level grouping is what hides NAT charges inside a generic bucket; usage type is where the truth lives.

# Substitute a whole billing month. Grouping by USAGE_TYPE is the
# point: "EC2 - Other" tells you nothing, "NatGateway-Bytes" does.
aws ce get-cost-and-usage 
  --time-period Start=YYYY-MM-01,End=YYYY-MM-01 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}'

Add S3 Storage Lens for the bucket. It tells you object count and average object size per prefix, which is the number that predicts your request bill and your Athena bill better than total storage does.

Lever 1: get the traffic off the NAT gateway

Usually the largest single saving available, and often a change you can make before lunch.

Start with the question nobody asks: does this function need to be in a VPC at all? A Lambda that calls the Salesforce REST API and writes to S3 touches nothing private. If it is in a VPC because that felt like the secure default, taking it out removes the NAT gateway from the path entirely and costs you nothing.

If it genuinely needs the VPC, because it also talks to RDS or something inside your network, add a gateway VPC endpoint for S3. AWS charges no hourly fee and no data processing fee for gateway endpoints, so S3 traffic simply stops flowing through NAT:

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

Two honest caveats. Gateway endpoints exist only for S3 and DynamoDB; everything else needs an interface endpoint, which has an hourly charge per availability zone plus a per-gigabyte charge. Still cheaper than NAT for AWS-service traffic, not free.

And the one specific to this workload: endpoints do nothing for your Salesforce traffic. Salesforce is not an AWS service, so those calls still traverse NAT. If you are pulling large volumes out of Salesforce from inside a VPC, the endpoint fixes the S3 half of the bill and leaves the other half untouched. That is the strongest argument for getting the function out of the VPC rather than papering over it.

Lever 2: fewer, bigger files

A sync running every five minutes across twelve Salesforce objects writes a lot of very small files. Storage for those is negligible. The costs are elsewhere and they compound.

• PUT requests are billed per request and cost meaningfully more than GETs. Millions of tiny writes is a request bill, not a storage bill.
• Athena opens every file it reads. Thousands of small objects per partition turn a fast scan into a slow, expensive one.
• Glue crawlers take longer over more objects, and they are billed by the second.

The fix is compaction: a scheduled job that rewrites each day’s partition into a small number of larger Parquet files and deletes the fragments. If you are on Iceberg, its compaction procedure does this for you and is the better reason to adopt it than any of the ones usually cited.

One trap worth flagging, because it is the opposite of what people expect. Lifecycle rules that move small objects to S3 Standard-IA can increase your bill: those classes have a minimum billable object size of 128 KB, so a 10 KB file is billed as 128 KB, plus a 30-day minimum duration. And S3 Intelligent-Tiering will not help either, since objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Compact first, then think about storage classes. In the other order you are paying more for the privilege.

Lever 3: log less

CloudWatch Logs is billed primarily on ingestion volume, and on a high-frequency pipeline it is routinely larger than the compute it is logging. A sync that logs every record it processes, running every five minutes, generates an astonishing amount of data whose only reader is a retention policy.

• Set a retention period on every log group. The default is to keep logs forever, and log groups created automatically by Lambda inherit it. This is the single most common unclaimed saving on an AWS account.
• Log per run, not per record. One structured summary line with counts, durations and outcome tells you more during an incident than ten thousand lines of per-record chatter.
• Turn off debug logging in production, and make the level an environment variable so raising it is a deliberate act rather than a redeploy.

Check the log classes available before you settle. There is a cheaper class aimed at logs you rarely query, which suits pipeline output well, with fewer features in exchange.

Lever 4: scan less

Athena bills on bytes scanned, so its cost is a property of your data layout, not your SQL. Three things move it, in order.

Parquet rather than JSON or CSV. Columnar means a query touching four columns reads four columns. This is usually a large multiple, not a percentage.

Partitions that match how people filter. If every query has a date range, partition by date. If nobody ever filters by region, partitioning by region just creates more small files.

Partition projection instead of a crawler. Athena works partitions out from the path pattern rather than reading them from the catalog, which removes the crawler cost, removes the repair step, and means partitions are never stale:

ALTER TABLE salesforce.opportunity SET TBLPROPERTIES (
  'projection.enabled'             = 'true',
  'projection.extract_date.type'   = 'date',
  'projection.extract_date.range'  = '2024-01-01,NOW',
  'projection.extract_date.format' = 'yyyy-MM-dd',
  'storage.location.template' =
    's3://acme-lake/raw/salesforce/opportunity/extract_date=${extract_date}/'
);

Set a per-query and per-workgroup data scanned limit while you are in there. It converts a runaway SELECT * from a surprise on the bill into an error someone sees immediately.

Lever 5: run it less often

The cheapest work is work you do not do. A sync running every five minutes because five minutes felt responsive, feeding a dashboard people open once each morning, is paying for freshness nobody consumes.

Ask what decision the data supports and how quickly it needs to be right. Hourly is enough for most reporting, and dropping from five minutes to hourly cuts invocations, requests, log volume and small-file creation all at once. It is the only lever here that improves four line items with one change.

Two false economies while you are looking. Cutting Lambda memory usually costs more, not less, because the function runs proportionally longer and you also lose CPU. And moving from Athena to a provisioned engine to escape per-query pricing only pays off at genuinely high query volume; below that you have swapped a variable cost for a fixed one and called it a saving.

Common mistakes

• Optimising compute because it is the part you understand, while data movement drives the bill.
• Reading Cost Explorer grouped by service, so NAT charges stay hidden inside “EC2 – Other”.
• Running a Lambda in a VPC when nothing it touches is private.
• Adding a gateway endpoint and expecting it to help traffic bound for Salesforce.
• Millions of small objects, and treating it as a storage problem rather than a request problem.
• Lifecycle rules that move sub-128 KB objects to an infrequent access class and increase the bill.
• No retention policy on log groups.
• Logging per record on a five-minute schedule.
• JSON in the lake because it was easier to write.
• Running a Glue crawler on a schedule when partition projection would do it for nothing.
• No cost allocation tags, so nobody can attribute anything.
• Syncing far more frequently than anybody reads the output.

Best practices

• Tag every resource the pipeline owns, and group Cost Explorer by usage type.
• Keep functions out of VPCs unless they need private resources.
• Gateway endpoints for S3 and DynamoDB on every VPC that touches them.
• Compact partitions on a schedule; measure average object size, not just total bytes.
• Compact before choosing storage classes, never the other way round.
• Retention on every log group, set at creation, enforced in your IaC.
• One structured summary log line per run.
• Parquet, partitioned the way people actually filter, with partition projection.
• Per-query and per-workgroup scan limits in Athena.
• Match the schedule to the decision the data supports.
• A budget alert on the pipeline’s tag, so the next surprise arrives as a notification.

FAQ

Why is “EC2 – Other” large when I run no EC2?

It is a catch-all that includes NAT gateway hours and data processing, EBS, elastic IPs and data transfer. Filter Cost Explorer to that service and group by usage type; if you see NAT gateway bytes near the top, that is your pipeline paying to move its own data.

Should my sync Lambda be in a VPC?

Only if it needs to reach something private, such as a database in your own subnets. Calling an external API and writing to S3 does not require it, and putting it in a VPC adds NAT charges and cold start complexity for no security benefit you can articulate.

Will Intelligent-Tiering fix my small files?

No. Objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Standard-IA is worse, because it has a 128 KB minimum billable size, so small objects are billed as if they were larger. Compaction is the fix; storage class is a decision for afterwards.

Glue or Lambda for the extraction?

Lambda for anything that fits in its execution limits, which covers most incremental CRM syncs, and it is cheaper for short frequent runs. Glue when you genuinely need Spark for volume or transformation. Glue billed by the second for a job that runs for ninety seconds is a poor trade.

What is the fastest saving to make?

Retention policies on log groups, then the S3 gateway endpoint. Both take minutes, neither carries risk, and between them they usually account for more than whatever you were planning to spend a week optimising.

The one thing to remember

Data pipeline bills are dominated by moving bytes, counting requests, ingesting logs and scanning storage. Compute is the part you think about and rarely the part you pay for.

So do the boring thing first: tag it, group Cost Explorer by usage type, and read the top five lines. The answer is almost always sitting there in a category you have been scrolling past, and it is usually cheaper to fix than to keep paying for.

Bill larger than it should be?

Pipeline cost work has a good ratio: a few days of investigation usually pays for itself in the first month. Work I take on:

• Cost breakdown of an existing Salesforce or CRM pipeline, by usage type, with a ranked list of what to fix and what it saves.
• Network path work: VPC endpoints, getting functions out of VPCs, and removing NAT from the data path.
• Storage layout: compaction jobs, partitioning strategy, Parquet or Iceberg conversion.
• Athena cost control: partition projection, workgroup scan limits, replacing scheduled crawlers.
• Logging and retention cleanup across accounts, which is usually the quickest win available.
• Cost allocation tagging and budget alerts so the next increase is noticed early.

Send me a month of Cost Explorer grouped by usage type and I will tell you where the money is going.

The post You Don’t Run Any EC2: Cutting the AWS Cost of Salesforce Data Pipelines appeared first on John Nessime.