“Salesforce handles it” is not an answer. Neither is “it’s in our VPC,” because it usually isn’t, and the parts that are in your VPC are the parts nobody has threat-modelled.

This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce’s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.

The boundaries a single request crosses

One user turn touches more systems than the diagrams suggest. Roughly:

1. Retrieval. The agent grounds itself in CRM records, Data Cloud objects and knowledge content.
2. Reasoning. The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.
3. Generation. A prompt is assembled and sent through the LLM gateway to a model provider.
4. Action. If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.

Steps one to three happen inside Salesforce’s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn’t. That sentence is the whole post, but the detail matters.

Retrieval: the part that’s better than you’d expect

Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.

The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.

Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.

Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.

The Trust Layer, and the default nobody expects

The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.

Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: LLM data masking is disabled for agents. Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.

The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <Person_0> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.

What this changes in practice:

• Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.
• If your compliance position was “no personal data leaves our boundary in identifiable form,” Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.
• Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.

None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.

Connecting Agentforce and AWS: four patterns

There are four ways these two systems talk, and they have different latency, failure and security characteristics.

Synchronous outbound: agent action calls AWS

A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.

Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.

The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.

Asynchronous outbound: Event Relay to EventBridge

Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.

This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.

Inbound: AWS calls Agentforce

The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent’s own logic.

Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.

MCP: tools rather than integrations

Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.

This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent’s context and by which your agent’s decisions reach someone else’s system. The registry exists precisely so that “which tools can this agent reach” is an administered answer rather than an emergent one. Use it as a control, not a catalogue.

Keeping the traffic private

By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.

What changes the moment you cross into AWS

This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.

• Zero retention becomes your policy, not Salesforce’s. If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.
• The audit trail splits. Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.
• User identity does not travel. This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.
• Model governance forks. A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.

Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce’s own workloads run there under Salesforce’s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.

Common mistakes

• Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog’s word for it, including this one.
• Logging full request payloads in Lambda during development and never turning it off.
• Giving the Lambda execution role broad data access because “the agent is trusted,” when the agent is a proxy for every user who can talk to it.
• Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.
• Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.
• Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.
• Deploying agents against a permissive sharing model and discovering the blast radius later.

Frequently asked questions

Does Agentforce data get used to train models?

Not on the Salesforce leg. Salesforce’s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.

Can I use my own Bedrock model inside Agentforce?

There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.

Sync or async for AWS integrations?

Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user’s turn, and they fail in front of the user rather than into a retry queue.

How do I stop the agent seeing data a user shouldn’t?

Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.

Do I need PrivateLink for this?

Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.

What should I log, and where?

On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.

The one thing to take away

Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day’s work.

What is hard is knowing precisely where Salesforce’s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.

Draw that line on your architecture diagram before the security review, not during it.

Need help designing or reviewing this?

I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:

• Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.
• Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.
• Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.
• Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.
• Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.
• Private Connect and PrivateLink setup where callouts should not touch the public internet.

If you have an integration already built, send me the agent action definition and the Lambda’s IAM policy. Those two together usually tell the whole story.

The post Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin appeared first on John Nessime.

The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands “broken”. The pipeline is faithfully recording something that is no longer true.

This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I’ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.

Why a Shopify sales dashboard with AWS drifts from the admin

Here’s the thing that catches almost everyone: a Shopify order is not an event, it’s a mutable record.

An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it’s exactly wrong for order data.

An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the orders/create payload and never revisits it, Monday’s revenue is frozen at the moment of checkout and it will only ever be too high.

This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.

The four adjustments that move historical numbers

• Refunds. Full or partial. A refund carries its own created_at, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.
• Order edits. A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.
• Cancellations. A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.
• Test and draft orders. Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.

The design consequence is simple to state and annoying to implement: your pipeline must be able to recompute any past day. Every storage and partitioning decision below follows from that one requirement.

Getting data out of Shopify: three paths, three trade-offs

Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you’re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you’re on borrowed time and you should plan the migration rather than discover the deadline.

Path 1: EventBridge partner event source

Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify’s own docs confirm EventBridge and Pub/Sub deliveries skip it.

You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the partner event source ARN, not the event bus ARN. That distinction accounts for a large share of the “I set it up and nothing arrives” threads on the Shopify forums.

A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you’ve seen the real shape of an event:

{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}

Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.

Path 2: HTTPS webhooks into API Gateway and Lambda

The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.

Verify against the raw request body, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn’t leak timing information:

import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -> bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)

Shopify sends the signature in the X-Shopify-Hmac-SHA256 header, base64-encoded, computed with your app’s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.

Acknowledge fast. Shopify’s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.

Path 3: scheduled GraphQL bulk pull

Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That’s a silent data loss mode with no local symptom at all.

So run a scheduled reconciliation pull alongside the stream. Shopify’s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That’s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.

A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.

Decide what “revenue” means before you write a line of SQL

This is where most reconciliation arguments actually live, and it isn’t an engineering problem at all until you’ve had the conversation.

Shopify’s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.

Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.

The currency trap

If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both shop_money and presentment_money: the amount in the store’s base currency, and the amount the customer actually saw and paid.

Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks “how much did we actually sell in Germany”, you’ll want it and it is painful to backfill.

One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.

Storage layout: partition by order date, never by arrival date

Two layers in S3. Keep them separate and keep them honest about what they are.

1. Raw. Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don’t delete it.
2. Curated. One row per order representing current state, in Parquet, partitioned by order date. This is what the dashboard queries. It is derived, disposable and rewritable.

The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today’s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.

Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you’re not using Firehose, extract the order date in your Lambda and write the prefix yourself.

Use partition projection so Athena stops guessing

The default Glue Data Catalog approach means running a crawler or issuing MSCK REPAIR TABLE to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.

Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:

CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);

Set the range start to your store’s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.

Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.

Net sales in one query

With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:

SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;

Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don’t.

Choosing the dashboard layer

Once the data is correct, this part is genuinely a preference. All of these work.

• Amazon QuickSight, now delivered as part of Amazon Quick Suite, is the least-friction option if you’re already in AWS. Its in-memory SPICE layer means viewers aren’t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.
• Grafana with the Athena data source is a good fit if you’re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you’d rather not run it.
• Power BI makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.
• Metabase or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.

The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.

Troubleshooting: symptom to cause

Totals are consistently higher than the Shopify admin

Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it’s refunds. If it’s a fixed offset on specific days, look for test orders or a QA run.

Totals are lower, and recent days are missing rows

Either partitions aren’t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.

Orders appear twice

Shopify’s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.

Numbers are right on the daily view, wrong on the monthly

Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.

Athena costs jumped without more data

Someone built a dashboard with a filter that doesn’t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.

Common mistakes

• Treating orders as immutable events and never revisiting a past day.
• Partitioning on arrival time because that’s the default, then discovering corrections are expensive.
• Verifying the HMAC against a re-serialised body instead of the raw bytes.
• Doing real work inside the webhook handler, generating duplicates under load.
• Summing presentment amounts across currencies.
• Registering the event bus ARN with Shopify instead of the partner event source ARN.
• Relying on webhooks alone with no scheduled reconciliation.
• Shipping a “revenue” number without ever defining what it includes.

Best practices

• Keep raw and curated layers separate. Raw is append-only; curated is rewritable.
• Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.
• Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.
• Run a nightly bulk pull over a rolling window and overwrite those partitions.
• Use partition projection. It removes an entire category of silent failure.
• Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.
• Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.
• Publish a reconciliation panel comparing your total to the admin’s for the same window. Surfacing the gap builds more trust than hiding it.

Frequently asked questions

Do I need a data warehouse, or is S3 and Athena enough?

For a single store’s order data, S3 with Athena is almost certainly enough, and it’s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you’re joining Shopify data against several other large sources, or when concurrent query load makes Athena’s queue times noticeable.

How near-real-time can this be?

Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.

Which webhook topics should I subscribe to?

At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they’re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.

Can I skip AWS and use a connector tool?

Yes, and for many stores that’s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You’re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn’t cover, or when row-based connector pricing outgrows the infrastructure cost.

How do I backfill historical orders?

Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you’ll spend an afternoon working out which one is lying.

What does a setup like this cost to run?

For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.

The one thing to take away

A Shopify sales dashboard with AWS doesn’t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.

Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.

Need help with your Shopify data pipeline on AWS?

I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:

• Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query
• Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job
• Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing
• Migrating REST Admin API integrations to GraphQL before the deadline forces the issue
• Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on
• Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned

If you’re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don’t match. It’s a much faster conversation than describing it in the abstract.

The post Shopify Sales Dashboard with AWS: Build One That Actually Reconciles appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

Eighteen months later there are forty flows, several of them on five-minute schedules because someone wanted fresher dashboards, and AppFlow is a line item that gets mentioned in the monthly review. Nothing was done wrong. The spreadsheet modelled the situation at the moment of the decision and that situation changed.

This is what makes an honest AppFlow vs Lambda vs MuleSoft cost comparison hard: you are not comparing three prices, you are comparing three different shapes, and the cheapest one at two integrations is frequently not the cheapest one at forty. The crossover is the entire decision, and almost nobody models it.

A note on numbers before we start. I am not quoting rate cards. AWS pricing varies by region and changes, MuleSoft publishes no list prices at all, and every comparison article with a neat table of dollar figures is quietly out of date. What does not change is the mechanism each one bills on, and that is what determines whether your bill grows with volume, with integration count, or not at all.

Three shapes

• AppFlow is pure consumption with no floor. You pay per flow run and per gigabyte processed. Two flows cost almost nothing; the bill scales with flows multiplied by frequency.
• Lambda is consumption too, but the meter is nearly irrelevant. The real cost is engineering time to build and own it, and the surrounding AWS services it drags along.
• MuleSoft is a floor. An annual contract sized on capacity or usage entitlements, paid whether you use it or not, largely independent of how many integrations you run inside it.

Draw those as lines against integration count and you get the whole argument. One starts near zero and climbs steeply. One starts moderate and climbs gently. One starts high and is close to flat. Where they cross depends entirely on your situation, which is why a generic answer is worthless and a framework is not.

AppFlow: cheap until the schedule tightens

AppFlow bills per successful flow run plus data processed. The detail that catches people, and it is stated plainly in AWS’s own pricing documentation, is this: a flow run that checks for new data and finds none still counts. You pay for polling, not for movement.

That turns schedule frequency into the dominant variable, and it is the one nobody models because at decision time everything is daily:

Flow runs per month, per flow:

  every 5 minutes   12 x 24 x 30  =  8,640
  every 15 minutes   4 x 24 x 30  =  2,880
  hourly                 24 x 30  =    720
  daily                       30  =     30

Twenty flows, five-minute schedule:  172,800 runs
The same twenty flows, hourly:        14,400 runs

Identical data. Twelve times the flow run charge.

Two more things that are not on the AppFlow line item. Standard S3 request and storage charges apply to whatever it writes, and KMS charges apply to the keys it uses. And on some connectors, running additional concurrent processes to speed a transfer is billed as additional flows.

Where AppFlow wins: a small number of standard SaaS-to-AWS movements on relaxed schedules, where nobody wants to own code. It is genuinely excellent at that, it goes live in an afternoon, and there is no floor to justify.

Where it stops winning: many flows, tight schedules, or transformation logic it cannot express. Schema drift handling is basic, and there is a point where you are working around the tool more than using it.

Lambda: the invoice is not the cost

For CRM-sized workloads, Lambda’s compute charge is close to a rounding error. A function that runs hourly for thirty seconds costs so little that optimising it is a waste of an afternoon.

The costs are elsewhere, and two of them are large.

The surrounding AWS services. NAT gateway data processing if the function sits in a VPC, CloudWatch Logs ingestion, S3 requests from small-file writes, Athena scans downstream. On a real pipeline these routinely exceed the compute charge several times over, and none of them appear under “Lambda” in Cost Explorer.

Engineering time, forever. Not just the build. Pagination, retries, watermarks, delete handling, schema drift, credential rotation, alerting, and someone available when it breaks at an awkward hour. This is the line that gets costed at zero in every comparison and is usually the largest number in the exercise.

The interesting property, though, is that the marginal cost of integration number eleven is much lower than integration number one, if you built a framework rather than eleven scripts. Shared extraction, shared state, shared error handling, one deployment pipeline. That is what makes the Lambda line climb gently instead of steeply. Teams that build eleven bespoke functions get the worst of both worlds: consumption pricing with none of the amortisation.

Where Lambda wins: you already have AWS engineers, you need control over the output shape, and the integration count is going up. Also anywhere the requirement is genuinely unusual, because a managed connector cannot express what it does not have a checkbox for.

Where it does not: a team of two who would rather ship product, or an organisation where the integration needs to be maintainable by people who are not engineers.

MuleSoft: a floor, and what it buys

MuleSoft publishes no list prices. Everything is quote-based, annual, and negotiated, which means any figure you find online is somebody else’s contract and tells you little about yours. It is also worth knowing that the commercial model has been repackaged relatively recently, moving new customers away from the older capacity-based structure toward usage entitlements, so a pricing article from a couple of years ago may describe a product you cannot buy.

Three structural things matter more than the number.

It is a commitment, not a meter. You size it up front and pay for it regardless of use, which is bad if you are unsure and fine if you are certain. Undersizing means overages; oversizing means paying for headroom.

Separate meters do not offset each other. Integration capacity, API traffic and add-on modules are metered independently, so spare room in one does not cover a spike in another. Model each axis separately or the renewal will be a surprise.

Renewal escalation is normal. Enterprise contracts typically carry annual uplift clauses, so the three-year total is meaningfully above three times year one. Negotiate that at signature, because the leverage is gone afterwards.

Now the fair part, because a post that treats MuleSoft as merely expensive is not being honest. What the floor actually buys: API management and governance as a first-class capability rather than something you assemble; a connector catalogue covering legacy and on-premise systems that neither AWS option touches; integrations that a trained analyst can maintain without a software team; audit trails, support contracts and SLAs that satisfy people who ask for them in writing. At a hundred integrations across several business units, the fixed cost is spread thin and the governance is the point.

The cost that is easy to miss is skills. DataWeave and the Anypoint tooling are proprietary, so you are hiring for a specific platform and your switching cost rises with every integration you build. That is not an argument against it; it is an argument for deciding deliberately rather than drifting in.

Where MuleSoft wins: many integrations, many teams, on-premise or legacy systems in the mix, and governance requirements with a compliance function attached to them.

Where it does not: three integrations, one team, all cloud. The floor will never amortise and you will spend the first year justifying it.

Modelling the crossover

Build the comparison over three years rather than one month, and project the inputs rather than freezing them.

1. Count integrations at year three, not today. Ask the people requesting them. The answer is always higher than the current number and it is the variable the whole model turns on.
2. Write down the schedule for each one, and defend it. “Every five minutes” is usually a preference, not a requirement. For AppFlow this is the difference between a small bill and a large one.
3. For AppFlow, multiply flows by runs per month. Apply current rates yourself, then add S3 requests and KMS.
4. For Lambda, cost the engineering explicitly. Build days plus ongoing maintenance days per year, at a loaded rate. If that number embarrasses you, it is probably the accurate one.
5. For MuleSoft, get an actual quote, model each meter separately, and include renewal escalation across the term.
6. Add the cost of being wrong. How much does switching cost in each direction? Consumption services you can walk away from. An annual contract and a proprietary skill set you cannot.

Then plot all three. You are looking for where the lines cross and whether your projected integration count sits comfortably on one side of it or uncomfortably near it. Near a crossover, pick the option that is cheaper to reverse.

Common mistakes

• Comparing one month at today’s integration count.
• Costing engineering time at zero on the build-it-yourself option.
• Not modelling flow run frequency, which is the single largest AppFlow variable.
• Forgetting that AppFlow charges for polling runs that find nothing.
• Attributing only the Lambda invoice to the Lambda option, ignoring NAT, logs and S3 requests.
• Building eleven bespoke Lambdas instead of one framework, so nothing amortises.
• Using published MuleSoft prices from an article, when no list prices exist and the packaging has changed.
• Ignoring renewal escalation when comparing an annual contract against consumption pricing.
• Treating a hybrid as a failure of nerve rather than the usual right answer.
• Choosing for the architecture you want rather than the team you have.

How I’d decide

Start with AppFlow if the integrations are standard, few, and not urgent. It is the lowest-commitment option and you can leave whenever you like.

Move to Lambda when you have AWS engineers, the count is climbing, and you need control over the output. Build it once as a framework, not repeatedly as scripts, or you get the cost profile without the benefit.

Consider MuleSoft when the problem is organisational rather than technical: many teams, legacy systems, governance requirements, and integrations that need to outlive the person who built them.

And be relaxed about mixing them. Plenty of sensible estates run AppFlow for the boring standard movements and custom code for the two integrations that are genuinely specific. Insisting on one tool for everything is a preference, not an architecture.

FAQ

Which is cheapest overall?

None of them, universally. AppFlow is cheapest at low integration counts and relaxed schedules. Lambda is cheapest at scale if you already have the engineers and build a shared framework. MuleSoft is cheapest only when the fixed cost is spread across many integrations and teams. The question is which shape fits your trajectory.

Why did my AppFlow bill grow when data volume didn’t?

Because you are billed per flow run, including runs that check for new data and find none. Somebody tightened a schedule, or the flow count grew. Both multiply the run charge without moving a single extra record.

Is Lambda really cheaper than AppFlow?

On the invoice, usually yes. In total cost, only once the engineering amortises across several integrations. For one or two simple flows, AppFlow is almost always the better economic answer, and the “we’ll just write a Lambda” instinct is a false economy at that scale.

Why can’t I find MuleSoft’s prices?

Because there aren’t any published. It is sold through quotes with negotiated discounts, so figures you find online are other people’s contracts. Get a quote, model each meter separately, and negotiate the renewal escalation before you sign rather than after.

Can I start on one and move later?

Between the two consumption options, fairly easily, especially if your destination schema is defined by you rather than by the tool. Off an annual contract with proprietary tooling, much less easily, which is exactly why the switching cost belongs in the model rather than as a footnote.

The one thing to remember

You are not choosing the cheapest tool. You are choosing a cost shape, and committing to how it behaves as your integration count and your schedules change over the next three years.

So model the trajectory, not the snapshot. Put engineering time in the spreadsheet at a real rate. And when the lines are close together, take the option you can walk away from, because being wrong cheaply is worth more than being right narrowly.

Need the model built for your numbers?

This decision usually gets made under time pressure with incomplete inputs, and then lived with for years. Work I take on:

• Building the three-year cost model for your actual integration inventory, schedules and volumes, with engineering time costed honestly.
• Auditing an existing AppFlow estate for flow count, schedule frequency and the charges that sit outside the AppFlow line item.
• Designing a shared extraction framework so custom integrations amortise instead of multiplying.
• Migrating between approaches without downtime, including keeping the destination schema stable so the choice stays reversible.
• Reviewing an integration platform contract for meter structure, overage terms and renewal escalation before you sign.
• Hybrid architectures where managed connectors handle the standard movements and code handles the awkward ones.

Send me your integration list with schedules and rough volumes, and I will tell you which shape you are actually buying.

The post Cheapest at Two Flows, Worst at Forty: AppFlow vs Lambda vs MuleSoft on Cost appeared first on John Nessime.

Then you check the invocation count and it is also zero, and has been since the day somebody disabled an EventBridge rule while cleaning up a different stack. The job has not run in three weeks. It never errored because it never started.

This is the shape of almost every integration monitoring failure: a healthy graph and a stopped job produce identical output. Zero errors is what success looks like and it is also what absence looks like, and if the only thing you measure is failure, the two are indistinguishable.

This is about how to monitor Salesforce integrations so that stopping is as loud as breaking. Four signals worth emitting, how to get them into CloudWatch cheaply, the alarm configuration that actually fires, and what belongs on a Grafana dashboard once you have them.

Four signals, not one

Most integration monitoring stops at errors and duration, because those come free from Lambda or your container platform. Both are worth having and neither answers the question anybody actually asks, which is “is the data right”.

The four that do:

• Liveness. Did it run at all?
• Volume. Did it move a plausible amount of data?
• Freshness. How old is the newest record on the destination side?
• Budget. How much of Salesforce’s daily API allowance have you spent?

Errors are a fifth, and the least interesting, because errors are the failure mode that already announces itself.

Liveness: the alarm that has to fire on silence

Emit a metric on every successful completion. A single count, value 1. Then alarm when it stops arriving.

The trap is in the CloudWatch defaults. TreatMissingData has four settings, and the default is missing, which sends the alarm to INSUFFICIENT_DATA when nothing arrives. That state is not ALARM. Nothing pages. Your dashboard shows a grey alarm that most people read as “fine”.

So heartbeat alarms need breaching. That much is standard advice. Here is the part that is not: even with breaching set, a heartbeat alarm can still fail to fire. CloudWatch evaluates over a range wider than your evaluation periods, and if it finds any real data point in that wider range, those override the missing ones. On a job that runs hourly, a successful run from earlier can keep the alarm quiet through several missed runs.

The robust version uses metric math to turn absence into a real zero, so there is no missing data to interpret:

aws cloudwatch put-metric-alarm 
  --alarm-name "opportunity-sync-not-running" 
  --alarm-description "No completed run in the last 90 minutes" 
  --comparison-operator LessThanThreshold 
  --threshold 1 
  --evaluation-periods 1 
  --treat-missing-data breaching 
  --alarm-actions "$SNS_TOPIC_ARN" 
  --metrics '[
    {
      "Id": "runs",
      "MetricStat": {
        "Metric": {
          "Namespace": "SalesforceSync",
          "MetricName": "RunCompleted",
          "Dimensions": [{"Name": "Integration", "Value": "opportunity-sync"}]
        },
        "Period": 5400,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "filled",
      "Expression": "FILL(runs, 0)",
      "ReturnData": true
    }
  ]'

FILL(runs, 0) substitutes a zero wherever the metric has no data point, so the alarm always has something real to compare against the threshold. The window is deliberately longer than the schedule: an hourly job gets ninety minutes, so one late run does not wake anybody.

Then do the thing everyone skips: disable the schedule in a test account and confirm the alarm actually goes red. An untested alarm is a belief, not a control.

Volume and freshness: is the data actually moving

A job can complete successfully and process nothing. A credential with the wrong field-level permissions, a filter that silently matches nothing, a watermark that got written before the data landed: all of these produce a clean run and an empty result.

Volume is the count of records read and written per run. Alarming on it is harder than liveness because the right number varies: a quiet Sunday legitimately looks like a broken Tuesday. Two approaches that work. Use CloudWatch anomaly detection, which learns the daily and weekly shape and alarms on departures from it. Or set a crude floor that only catches the catastrophic case, which is usually zero, and accept that you will not catch a fifty percent drop.

I would start with the crude floor. It is five minutes of work and catches the failure that actually happens.

Freshness is the better metric and almost nobody emits it. At the end of each run, query the destination for the newest record’s modified timestamp, subtract it from now, and publish the difference in seconds. That single number answers the business question directly: how far behind Salesforce are we right now?

It also collapses several failure modes into one signal. A stopped job, a job that runs but writes nothing, a job stuck retrying, a job silently filtered down to zero rows: all of them show up as lag climbing. If you only add one metric from this post, add this one.

Budget: watch the Salesforce allowance

Your org has a daily API allowance shared across every integration touching it. Exceeding it does not just break your sync; it breaks marketing automation, support tooling, and whatever else somebody connected two years ago.

Enforcement is initially soft, and then it is not: past a protection threshold, calls come back as 403 with REQUEST_LIMIT_EXCEEDED until the rolling window drains. By that point you are in an incident that spans several teams.

The good news is that this costs nothing to observe. Salesforce returns your current consumption on ordinary REST responses in a header, so you get it on calls you were making anyway:

# Sforce-Limit-Info: api-usage=1212/15000
#
# Free: no extra API call, which matters when the thing you are
# measuring is an API budget. Add real error handling before
# shipping this; the header is not guaranteed on every response.
raw = response.headers.get("Sforce-Limit-Info", "")
used, allowed = (int(v) for v in raw.split("api-usage=")[1].split("/"))
emit("ApiUsagePercent", used / allowed * 100)

For a fuller picture, the /services/data/vXX.X/limits endpoint returns every allocation in the org, including DailyApiRequests with its max and remaining values. It needs the View Setup and Configuration permission and the numbers lag by a few minutes. Poll it on a schedule rather than per request, and alarm on percentage consumed rather than absolute calls, so the alarm survives a licence change.

Graph consumption by integration if you can attribute it. The conversation about which team is burning the allowance goes very differently when there is a chart.

Getting the metrics in without a bill shock

You can call PutMetricData directly, and it works, and it is a synchronous API call in the hot path of your job that can fail or add latency. Custom metrics are also charged per metric per month, and a metric is every unique combination of name and dimensions, so a dimension with high cardinality gets expensive quietly.

The better default is Embedded Metric Format: write structured JSON to stdout and CloudWatch extracts the metrics from your logs. No API call, no added latency, and the log line stays queryable in Logs Insights alongside the metric.

{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [{
      "Namespace": "SalesforceSync",
      "Dimensions": [["Integration"]],
      "Metrics": [
        { "Name": "RunCompleted",     "Unit": "Count"   },
        { "Name": "RecordsWritten",   "Unit": "Count"   },
        { "Name": "SourceLagSeconds", "Unit": "Seconds" },
        { "Name": "ApiUsagePercent",  "Unit": "Percent" }
      ]
    }]
  },
  "Integration": "opportunity-sync",
  "RunId": "a41c9f",
  "RunCompleted": 1,
  "RecordsWritten": 4127,
  "SourceLagSeconds": 312,
  "ApiUsagePercent": 8.1
}

Note what is a dimension and what is not. Integration is a dimension because it has a handful of values and you want to alarm per integration. RunId is a plain field: searchable in the logs, and not a dimension, because making it one would create a new metric on every run. That distinction is the whole cost story.

Grafana on top

CloudWatch dashboards are fine and Grafana is better for this, for three reasons: you can put Salesforce metrics next to your warehouse and application metrics on one screen, the alerting is more expressive, and non-engineers will actually open it.

Add CloudWatch as a data source using an IAM role rather than access keys, scoped to cloudwatch:GetMetricData, cloudwatch:ListMetrics and the Logs Insights permissions if you want log panels. One honest cost note: Grafana queries CloudWatch through the metric data API, which is billed per metric requested, so a busy dashboard on a short refresh interval is a real line item. Set a sane refresh, avoid auto-refresh on wall displays, and use the caching in Grafana’s CloudWatch data source.

What goes on the dashboard, in order down the page:

• Freshness per integration, as a stat panel with thresholds. This is the panel people look at.
• Time since last successful run, per integration.
• Records processed, over a window long enough to show the weekly shape.
• API allowance consumed, as a percentage with a threshold line.
• Errors and duration, at the bottom, where they belong.

One dashboard, one screen, no scrolling. A dashboard nobody can read at a glance during an incident is decoration.

Alerts people don’t ignore

Decide deliberately where alerting lives. CloudWatch alarms are more reliable, because they keep working when Grafana is down, and Grafana alerts are more flexible and can span data sources. My default is CloudWatch for the small number of alerts that page someone, and Grafana for everything informational.

Three things that separate a useful alert from noise. Alarm on the symptom, not the cause: “Opportunity data is more than two hours stale” is actionable in a way “Lambda errors greater than zero” is not. Put the runbook link in the alarm description, since that field ends up in the notification and is the only documentation anybody reads at midnight. And use composite alarms to suppress the cascade, so a Salesforce outage produces one page rather than nine.

Common mistakes

• Monitoring only errors, so a stopped job looks identical to a healthy one.
• Leaving TreatMissingData at its default on a heartbeat alarm.
• Setting it to breaching and assuming that is sufficient, without handling the evaluation range.
• Never testing that an alarm fires by actually breaking something.
• No freshness metric, so nobody can answer how far behind the data is.
• Ignoring API allowance until an integration you do not own breaks.
• High-cardinality dimensions such as record ID or run ID, and the bill that follows.
• Calling PutMetricData synchronously in the job’s critical path.
• Alerting on causes rather than on user-visible symptoms.
• A dashboard that requires scrolling and interpretation during an incident.

Best practices

• Emit liveness, volume, freshness and API budget from every integration, as a standard.
• Heartbeat alarms with breaching plus FILL(), and a window longer than the schedule.
• Freshness as the headline metric, because it maps to a question the business asks.
• Embedded Metric Format rather than direct API calls.
• Low-cardinality dimensions; everything else stays a log field.
• Alarm on percentage of the API allowance, not absolute calls.
• IAM roles for the Grafana data source, and a refresh interval you have costed.
• Runbook links in alarm descriptions.
• Composite alarms to collapse cascades into one page.
• A quarterly test that breaks each integration on purpose and confirms someone gets told.

FAQ

Why didn’t my CloudWatch alarm fire when the job stopped?

Almost certainly TreatMissingData. The default sends the alarm to INSUFFICIENT_DATA, which is not ALARM and pages nobody. Set it to breaching, and wrap the metric in FILL() so there is no missing data for CloudWatch to reinterpret.

What’s the single most useful metric to add?

Freshness: how old the newest record on the destination side is. It catches stopped jobs, empty runs, stuck retries and silent filtering with one number, and it is the only one of these metrics a non-engineer can interpret.

CloudWatch dashboards or Grafana?

Grafana if you already run it, because you can put Salesforce, warehouse and application metrics on one screen. CloudWatch if you do not, because a second system to operate is not free. Either way keep the paging alarms in CloudWatch so they survive Grafana being down.

Will custom metrics be expensive?

Only if you make them so. Cost scales with unique name-and-dimension combinations, so a handful of metrics dimensioned by integration name is negligible. Adding a run ID or record ID as a dimension is how the bill grows without anyone noticing.

How do I monitor a third-party connector I can’t add code to?

Monitor the destination instead. A scheduled job that queries the target for the newest record’s timestamp and emits it as a freshness metric works regardless of what wrote the data, and it is arguably a better test because it measures the outcome rather than the process.

The one thing to remember

Absence of failure is not evidence of success. An integration that stopped produces exactly the same error graph as one working perfectly, and every default in your monitoring stack is tuned to stay quiet when data stops arriving rather than to shout about it.

So measure the thing you actually care about. Not “did it error” but “how stale is the data right now”, alarmed in a way that fires on silence, and tested by deliberately breaking it. Everything else on the dashboard is supporting evidence.

Want this built properly?

Integration monitoring tends to get added after the first silent failure, which is one failure too late. Work I take on:

• Instrumenting Salesforce integrations with liveness, volume, freshness and API budget metrics via CloudWatch.
• Auditing existing alarms for the ones that cannot fire, and fixing the missing-data handling.
• Building the Grafana dashboard and data source, including cost-aware query and refresh configuration.
• Alert design: symptom-based alarms, composite alarms to suppress cascades, runbooks attached where people will read them.
• API allowance monitoring and attribution across multiple integrations sharing one org.
• Running a failure drill so you know the alerting works before you need it.

Tell me how you would currently find out that a sync stopped, and I will tell you how long it would take.

The post Zero Errors, Zero Records: Monitoring Salesforce Integrations with CloudWatch and Grafana appeared first on John Nessime.

“EC2 – Other” is where AWS files NAT gateway charges, along with EBS, elastic IPs and assorted data transfer. Your Salesforce sync runs in a private subnet, so every byte it pulls from Salesforce and every byte it writes to S3 goes through a NAT gateway, which bills an hourly rate plus a charge per gigabyte processed. None of that appears under Lambda, or under S3, or under the tag you put on the pipeline.

That is the recurring theme when you look at the AWS cost of Salesforce data pipelines: the spend is almost never in the place you have been optimising. People tune Lambda memory settings while the bill is being driven by data movement, request counts, log ingestion and scanned bytes.

Five levers, in the order I would pull them.

First, find out where it actually goes

Every hour spent guessing is an hour not spent on the line item that matters. Tag everything the pipeline owns with a consistent cost allocation tag, activate it in the billing console, and wait for it to backfill.

Then group Cost Explorer by usage type rather than by service. Service-level grouping is what hides NAT charges inside a generic bucket; usage type is where the truth lives.

# Substitute a whole billing month. Grouping by USAGE_TYPE is the
# point: "EC2 - Other" tells you nothing, "NatGateway-Bytes" does.
aws ce get-cost-and-usage 
  --time-period Start=YYYY-MM-01,End=YYYY-MM-01 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}'

Add S3 Storage Lens for the bucket. It tells you object count and average object size per prefix, which is the number that predicts your request bill and your Athena bill better than total storage does.

Lever 1: get the traffic off the NAT gateway

Usually the largest single saving available, and often a change you can make before lunch.

Start with the question nobody asks: does this function need to be in a VPC at all? A Lambda that calls the Salesforce REST API and writes to S3 touches nothing private. If it is in a VPC because that felt like the secure default, taking it out removes the NAT gateway from the path entirely and costs you nothing.

If it genuinely needs the VPC, because it also talks to RDS or something inside your network, add a gateway VPC endpoint for S3. AWS charges no hourly fee and no data processing fee for gateway endpoints, so S3 traffic simply stops flowing through NAT:

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

Two honest caveats. Gateway endpoints exist only for S3 and DynamoDB; everything else needs an interface endpoint, which has an hourly charge per availability zone plus a per-gigabyte charge. Still cheaper than NAT for AWS-service traffic, not free.

And the one specific to this workload: endpoints do nothing for your Salesforce traffic. Salesforce is not an AWS service, so those calls still traverse NAT. If you are pulling large volumes out of Salesforce from inside a VPC, the endpoint fixes the S3 half of the bill and leaves the other half untouched. That is the strongest argument for getting the function out of the VPC rather than papering over it.

Lever 2: fewer, bigger files

A sync running every five minutes across twelve Salesforce objects writes a lot of very small files. Storage for those is negligible. The costs are elsewhere and they compound.

• PUT requests are billed per request and cost meaningfully more than GETs. Millions of tiny writes is a request bill, not a storage bill.
• Athena opens every file it reads. Thousands of small objects per partition turn a fast scan into a slow, expensive one.
• Glue crawlers take longer over more objects, and they are billed by the second.

The fix is compaction: a scheduled job that rewrites each day’s partition into a small number of larger Parquet files and deletes the fragments. If you are on Iceberg, its compaction procedure does this for you and is the better reason to adopt it than any of the ones usually cited.

One trap worth flagging, because it is the opposite of what people expect. Lifecycle rules that move small objects to S3 Standard-IA can increase your bill: those classes have a minimum billable object size of 128 KB, so a 10 KB file is billed as 128 KB, plus a 30-day minimum duration. And S3 Intelligent-Tiering will not help either, since objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Compact first, then think about storage classes. In the other order you are paying more for the privilege.

Lever 3: log less

CloudWatch Logs is billed primarily on ingestion volume, and on a high-frequency pipeline it is routinely larger than the compute it is logging. A sync that logs every record it processes, running every five minutes, generates an astonishing amount of data whose only reader is a retention policy.

• Set a retention period on every log group. The default is to keep logs forever, and log groups created automatically by Lambda inherit it. This is the single most common unclaimed saving on an AWS account.
• Log per run, not per record. One structured summary line with counts, durations and outcome tells you more during an incident than ten thousand lines of per-record chatter.
• Turn off debug logging in production, and make the level an environment variable so raising it is a deliberate act rather than a redeploy.

Check the log classes available before you settle. There is a cheaper class aimed at logs you rarely query, which suits pipeline output well, with fewer features in exchange.

Lever 4: scan less

Athena bills on bytes scanned, so its cost is a property of your data layout, not your SQL. Three things move it, in order.

Parquet rather than JSON or CSV. Columnar means a query touching four columns reads four columns. This is usually a large multiple, not a percentage.

Partitions that match how people filter. If every query has a date range, partition by date. If nobody ever filters by region, partitioning by region just creates more small files.

Partition projection instead of a crawler. Athena works partitions out from the path pattern rather than reading them from the catalog, which removes the crawler cost, removes the repair step, and means partitions are never stale:

ALTER TABLE salesforce.opportunity SET TBLPROPERTIES (
  'projection.enabled'             = 'true',
  'projection.extract_date.type'   = 'date',
  'projection.extract_date.range'  = '2024-01-01,NOW',
  'projection.extract_date.format' = 'yyyy-MM-dd',
  'storage.location.template' =
    's3://acme-lake/raw/salesforce/opportunity/extract_date=${extract_date}/'
);

Set a per-query and per-workgroup data scanned limit while you are in there. It converts a runaway SELECT * from a surprise on the bill into an error someone sees immediately.

Lever 5: run it less often

The cheapest work is work you do not do. A sync running every five minutes because five minutes felt responsive, feeding a dashboard people open once each morning, is paying for freshness nobody consumes.

Ask what decision the data supports and how quickly it needs to be right. Hourly is enough for most reporting, and dropping from five minutes to hourly cuts invocations, requests, log volume and small-file creation all at once. It is the only lever here that improves four line items with one change.

Two false economies while you are looking. Cutting Lambda memory usually costs more, not less, because the function runs proportionally longer and you also lose CPU. And moving from Athena to a provisioned engine to escape per-query pricing only pays off at genuinely high query volume; below that you have swapped a variable cost for a fixed one and called it a saving.

Common mistakes

• Optimising compute because it is the part you understand, while data movement drives the bill.
• Reading Cost Explorer grouped by service, so NAT charges stay hidden inside “EC2 – Other”.
• Running a Lambda in a VPC when nothing it touches is private.
• Adding a gateway endpoint and expecting it to help traffic bound for Salesforce.
• Millions of small objects, and treating it as a storage problem rather than a request problem.
• Lifecycle rules that move sub-128 KB objects to an infrequent access class and increase the bill.
• No retention policy on log groups.
• Logging per record on a five-minute schedule.
• JSON in the lake because it was easier to write.
• Running a Glue crawler on a schedule when partition projection would do it for nothing.
• No cost allocation tags, so nobody can attribute anything.
• Syncing far more frequently than anybody reads the output.

Best practices

• Tag every resource the pipeline owns, and group Cost Explorer by usage type.
• Keep functions out of VPCs unless they need private resources.
• Gateway endpoints for S3 and DynamoDB on every VPC that touches them.
• Compact partitions on a schedule; measure average object size, not just total bytes.
• Compact before choosing storage classes, never the other way round.
• Retention on every log group, set at creation, enforced in your IaC.
• One structured summary log line per run.
• Parquet, partitioned the way people actually filter, with partition projection.
• Per-query and per-workgroup scan limits in Athena.
• Match the schedule to the decision the data supports.
• A budget alert on the pipeline’s tag, so the next surprise arrives as a notification.

FAQ

Why is “EC2 – Other” large when I run no EC2?

It is a catch-all that includes NAT gateway hours and data processing, EBS, elastic IPs and data transfer. Filter Cost Explorer to that service and group by usage type; if you see NAT gateway bytes near the top, that is your pipeline paying to move its own data.

Should my sync Lambda be in a VPC?

Only if it needs to reach something private, such as a database in your own subnets. Calling an external API and writing to S3 does not require it, and putting it in a VPC adds NAT charges and cold start complexity for no security benefit you can articulate.

Will Intelligent-Tiering fix my small files?

No. Objects under 128 KB are never auto-tiered and stay at Frequent Access rates. Standard-IA is worse, because it has a 128 KB minimum billable size, so small objects are billed as if they were larger. Compaction is the fix; storage class is a decision for afterwards.

Glue or Lambda for the extraction?

Lambda for anything that fits in its execution limits, which covers most incremental CRM syncs, and it is cheaper for short frequent runs. Glue when you genuinely need Spark for volume or transformation. Glue billed by the second for a job that runs for ninety seconds is a poor trade.

What is the fastest saving to make?

Retention policies on log groups, then the S3 gateway endpoint. Both take minutes, neither carries risk, and between them they usually account for more than whatever you were planning to spend a week optimising.

The one thing to remember

Data pipeline bills are dominated by moving bytes, counting requests, ingesting logs and scanning storage. Compute is the part you think about and rarely the part you pay for.

So do the boring thing first: tag it, group Cost Explorer by usage type, and read the top five lines. The answer is almost always sitting there in a category you have been scrolling past, and it is usually cheaper to fix than to keep paying for.

Bill larger than it should be?

Pipeline cost work has a good ratio: a few days of investigation usually pays for itself in the first month. Work I take on:

• Cost breakdown of an existing Salesforce or CRM pipeline, by usage type, with a ranked list of what to fix and what it saves.
• Network path work: VPC endpoints, getting functions out of VPCs, and removing NAT from the data path.
• Storage layout: compaction jobs, partitioning strategy, Parquet or Iceberg conversion.
• Athena cost control: partition projection, workgroup scan limits, replacing scheduled crawlers.
• Logging and retention cleanup across accounts, which is usually the quickest win available.
• Cost allocation tagging and budget alerts so the next increase is noticed early.

Send me a month of Cost Explorer grouped by usage type and I will tell you where the money is going.

The post You Don’t Run Any EC2: Cutting the AWS Cost of Salesforce Data Pipelines appeared first on John Nessime.