Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.

This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It’s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.

Failure family one: the join key is not the email address

Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.

Start with where identity actually lives in each system:

• Shopify has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.
• Stripe has a customer ID, but only if a Customer object was created. Checkout Sessions that don’t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.
• The CRM keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.
• The helpdesk keys on the requester’s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.

Here’s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look better than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.

Build an identifier table, not a fuzzy match

The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.

-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);

Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail’s dot and plus handling. Stripping +tags is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.

Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.

One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into metadata. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That’s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.

Failure family two: webhooks arrive twice, out of order, or not at all

Both platforms are explicit about this, and both get treated as if they weren’t.

Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event id (the one starting evt_) stays identical across every retry, which makes it your deduplication key. Shopify sends X-Shopify-Webhook-Id for the same purpose, signs the raw body into X-Shopify-Hmac-Sha256, and states plainly that ordering isn’t guaranteed within a topic or across topics for the same resource. Shopify’s own documentation goes further and tells you not to rely on webhooks as your only source of data.

For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a customers/update that changes someone’s email arriving before the orders/create that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn’t exist yet, or worse, to the wrong one. Order your writes by the event’s own timestamp, using X-Shopify-Triggered-At or the updated_at in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.

Do almost nothing in the handler

Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That’s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.

Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:

-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;

Return a 2xx for a duplicate. A duplicate is a success from the sender’s point of view, and returning an error just schedules more of them.

Signature verification has one recurring trap on the Shopify side: it runs against the raw body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.

import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -> bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)

Worth knowing before you spend an afternoon on it: after rotating an app’s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that’s usually why.

And because delivery isn’t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it’s the one people skip. If you’d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.

Failure family three: “revenue” means four different things

This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they’re measuring different things and nobody wrote down which.

• Shopify reports gross sales at the moment the order is placed, in the store’s currency and the store’s timezone.
• Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.
• A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.
• A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.

There’s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe’s processing infrastructure, but it’s Shopify’s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.

Do not build a single revenue column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.

Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of “the number moved and nobody changed anything.”

On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you’re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.

Failure family four: deletion is the requirement nobody models

If you’re building this as a Shopify app, three compliance webhooks are mandatory: customers/data_request, customers/redact and shop/redact. You acknowledge with a 200 and complete the action within thirty days. shop/redact fires forty-eight hours after uninstall and covers the shop’s data. These are enforced at app review, so you’ll find out about them either way.

The interesting problem isn’t receiving the webhook. It’s that by the time it arrives, that person’s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.

The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries person_key and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.

Practical pieces that make it real:

• Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.
• Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.
• Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.
• Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can’t perform.

Failure family five: pushing the unified view back out

A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.

That’s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor’s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you’ll be running several vendors plus warehouse compute plus the people to maintain the models.

Three things go wrong here more or less every time.

The write loop. You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.

The resync stampede. Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.

Syncing logic instead of keys. Push person_key into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.

A pipeline shape that holds up

Put together, the order of operations matters more than the tool choice:

1. Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.
2. Deduplicate on the provider’s event ID, atomically, with a retention window longer than the provider’s retry window.
3. Normalize identifiers into the identifier table as rows arrive, with the source recorded.
4. Resolve identity deterministically into person_key, as a scheduled job, never inside a request handler.
5. Build fact tables per system at their own grain, keyed on person_key, with no raw PII.
6. Model the customer view on top, in dbt or whatever your team already uses. The Fivetran dbt_customer360 package is a reasonable starting point if you want to see how someone else structured the identity step.
7. Activate a small, named, namespaced set of fields back into the CRM and helpdesk.
8. Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.

Troubleshooting: symptom to root cause

These are the ones that come up repeatedly, and the check that usually settles them.

• Warehouse customer count is much higher than Shopify’s. Guest orders are minting a new person on every purchase. Check how many person rows have exactly one identifier and one order.
• Warehouse count is much lower. An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.
• BI revenue doesn’t match Stripe payouts. Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.
• A CRM field keeps flipping between two values. Write loop. Check whether the field’s namespace is excluded from your ELT source configuration.
• Recent orders missing, older ones fine. Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.
• Every Shopify signature suddenly fails. Either a body parser is running before verification, or the client secret was rotated within the last hour.
• Tickets attached to the wrong customer after a merge. The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.

Common mistakes

• Joining on email in the model layer instead of resolving identity once, upstream.
• Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.
• Doing identity resolution, enrichment or CRM writes inside the webhook handler.
• Ordering profile updates by arrival time rather than by the event’s own timestamp.
• One revenue column that finance and marketing are both told to use.
• Storing amounts as floats, or recalculating historical FX from a rates table.
• Copying PII into every downstream table because it was convenient at the time.
• Reaching for probabilistic matching before the deterministic keys have been exhausted.
• Assuming a Shopify store on Shopify Payments has a Stripe account you can query.

Best practices worth the effort

• Own the surrogate key. Never let a vendor’s internal ID become the primary identity of a person in your systems.
• Write your person_key into Stripe metadata and Shopify customer metafields at creation time, so future joins are exact rather than inferred.
• Keep raw payloads immutable and replayable. Every model bug you’ll ever have is cheaper to fix if you can reprocess from raw.
• Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.
• Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.
• Reconcile one payout end to end by hand before you trust the automated version. It’s the fastest way to find a grain error.
• Document the metric definitions next to the models. Most “the data is wrong” tickets are definition disputes.

FAQ

Do I need a CDP to unify Shopify, Stripe, CRM and support data?

Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.

How do I handle guest checkout customers?

Create a person from the order’s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.

Should I sync everything into the CRM or keep it in the warehouse?

Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.

Why don’t my Shopify sales match my Stripe payouts?

Because they’re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.

Can I use email as the primary key if my store requires account creation?

You can get away with it longer, but it’s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.

How do I test the identity resolution before trusting it?

Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.

What’s the minimum viable version of this?

Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That’s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.

Conclusion: unify Shopify, Stripe, CRM and support data around a key you own

If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer is, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.

Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.

Need help wiring this together?

I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:

• Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.
• Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.
• Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.
• Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.
• Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.
• Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.

If you’re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It’s much easier to say something useful about a real artifact than about a diagram.

The post One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data appeared first on John Nessime.

That is the failure mode that defines this problem. An insurance claims processing pipeline on AWS almost never falls over in the way you designed it to fall over. It succeeds. It emits well-formed JSON. It hands a number to a payment system, and the number is wrong, and nothing in the pipeline had any reason to think otherwise.

This post is organized by failure family rather than by service. I’ll walk through the six ways these pipelines go quietly wrong, what each one costs, and what the fix actually looks like in Textract, Bedrock, Step Functions and S3. There’s a troubleshooting section, the mistakes I see repeated, and an FAQ at the end.

The shape most claims pipelines end up with

Before the failure families make sense, the skeleton. Almost every serverless claims pipeline lands on roughly the same set of stages, whatever the vendor deck calls them:

1. Intake. A document lands in S3 from a portal upload, an SFTP drop, or a mail scanning vendor. An S3 event or EventBridge rule starts an execution.
2. Classification. Work out what the packet actually contains. A first notice of loss, a CMS-style claim form, a police report, an itemized bill, forty pages of photographs.
3. Extraction. Pull the fields you need. Amazon Textract for OCR, forms and tables, or Amazon Bedrock Data Automation with a blueprint that names the fields directly.
4. Validation. Check the extracted values against business rules, policy data, and each other.
5. Routing. Straight-through processing, human review, or rejection with a reason.
6. Persistence and audit. The claim record, the extraction artifacts, and enough evidence to explain a decision months later.

Nothing controversial there. AWS publishes an open-source GenAI IDP Accelerator that implements exactly this shape, with a Bedrock Data Automation mode and a Textract-plus-foundation-model pipeline mode, and it’s a reasonable place to start reading. The interesting part is not the boxes. It’s what happens between them.

Failure family one: the field that was never there

This is the one that pays out the wrong number, and it is worth more attention than everything else in this post combined.

Every extraction service gives you confidence scores. So the obvious design is a gate: if every field scores above some threshold, approve automatically; if anything falls below, send it to a human. That gate is sound reasoning applied to the wrong population.

A confidence score only exists for a value that came back. When the extractor doesn’t find a field at all, there is no low score to catch, because there’s nothing to score. The gate iterates over four returned fields, finds all four above threshold, and reports a clean pass. The fifth field, the one that determines coordination of benefits or the deductible offset, is simply absent from the response. Downstream code treats absent as zero, or as null, or as “not applicable,” and the claim goes through.

The fix is structural, not statistical. Validate presence against a schema before you validate confidence, and treat the two as separate gates with separate outcomes:

# Two gates, not one. Presence first, then confidence.
# 'extracted' is the flattened field map from Textract Queries
# or a Bedrock Data Automation blueprint result.

REQUIRED = {
    "claim_number",
    "date_of_service",
    "billed_amount",
    "member_id",
    "secondary_payer_indicator",
}

def gate(extracted, scores, threshold=0.95):
    missing = REQUIRED - set(extracted)
    if missing:
        # Never silently default. This is a routing decision.
        return "HUMAN_REVIEW", {"reason": "missing_fields",
                                "fields": sorted(missing)}

    weak = [f for f in REQUIRED if scores.get(f, 0.0) < threshold]
    if weak:
        return "HUMAN_REVIEW", {"reason": "low_confidence",
                                "fields": sorted(weak)}

    return "STRAIGHT_THROUGH", {}

Two details matter here. The set difference is computed against a declared schema, not against whatever keys happen to be in the response, so an absent field becomes a first-class routing reason. And scores.get(f, 0.0) defaults to zero rather than to a passing value, so a field that arrives without a score fails closed.

If you’re on Textract Queries, there’s a second reason to be explicit: Queries let you attach an alias to each question, which means your schema keys are yours rather than whatever label happened to be printed on the form. That’s the difference between “the field is missing” and “the field moved and we didn’t notice.”

Failure family two: confidence scores that answer a different question

Assume you’ve fixed presence. The next trap is what the confidence number is measuring.

OCR confidence is a statement about characters. It says the model is highly sure those pixels read 1,240.00. It is not a statement that 1,240.00 is the billed amount rather than the allowed amount from the box directly above it, or the prior balance from a remittance summary that happened to be stapled into the same packet. Read it as a legibility score, because that’s closer to what it is.

Bedrock Data Automation narrows this gap: blueprints define fields semantically, confidence scores come with bounding boxes, and the visual grounding lets you point at the region a value came from. Textract Queries narrow it too, by asking a question rather than harvesting a label. Neither eliminates the problem, because a high-confidence read of the wrong region still scores high.

What actually catches this is cross-field invariants. They cost almost nothing and they fail for reasons a human can read:

• Arithmetic. Line items sum to the claimed total. If they don’t, one of the two is wrong and you don’t yet know which.
• Temporal. Date of service falls inside the policy period and before the date of submission. A service date after the submission date is a parsing error nine times out of ten.
• Referential. The member or policy identifier resolves against your system of record. An identifier that matches the format but not a real record is a strong signal you read the wrong box.
• Range. Amounts within a plausible band for the claim type. Not a fraud model, just a tripwire for a decimal point that moved.
• Page provenance. Fields that must come from the same page or the same document within the packet. Bounding box data makes this checkable rather than assumed.

An invariant failure is more useful than a low score, because it names a contradiction. “Line items sum to 1,180 but the claimed total reads 1,240” is something a reviewer resolves in seconds. “Confidence 0.91” is something a reviewer stares at.

Failure family three: the claim that stops halfway

Claims documents are multi-page packets, so you’ll be using Textract’s asynchronous operations. That means jobs, notifications, and a whole class of orchestration bugs that only show up under load or after a weekend.

The asynchronous pattern is: call StartDocumentAnalysis, get a JobId back, and let Textract publish completion to an SNS topic you nominate. A request looks like this:

{
  "DocumentLocation": {
    "S3Object": { "Bucket": "claims-intake", "Name": "packets/abc123.pdf" }
  },
  "FeatureTypes": ["FORMS", "TABLES"],
  "ClientRequestToken": "abc123-v1",
  "JobTag": "fnol-packet",
  "NotificationChannel": {
    "SNSTopicArn": "arn:aws:sns:REGION:ACCOUNT:textract-complete",
    "RoleArn": "arn:aws:iam::ACCOUNT:role/TextractPublishRole"
  },
  "OutputConfig": {
    "S3Bucket": "claims-extraction",
    "S3Prefix": "raw/"
  },
  "KMSKeyId": "alias/claims-cmk"
}

Three of those parameters are doing load-bearing work that is easy to skip.

ClientRequestToken is the idempotency token. Reuse the same token and you get the same JobId back instead of a second job. Derive it from the document, not from the invocation, and a Lambda retry or a duplicated S3 event stops turning into a duplicate charge and a duplicate claim record.

OutputConfig writes results into a bucket you control. Without it, results stay internal to Textract and the only way to read them is the Get operations, which have their own throttling limits. Under concurrency those limits become the bottleneck: you end up polling more jobs than you’re allowed to poll, backing off, and watching end-to-end latency climb for reasons that have nothing to do with the documents. Writing to S3 sidesteps the whole path.

JobTag shows up in the completion notification. In a mixed pipeline where the same topic carries first notice of loss packets, itemized bills and ID documents, that tag is what lets the notification handler route without a lookup.

One expiry to plan around: a Textract JobId is only valid for seven days. If your retry story is “requeue it and someone will look on Monday,” a bad weekend turns recoverable failures into full reprocessing. Persist the S3 output location, not the job identifier.

Callbacks that never come back

Human review means pausing a workflow for hours or days, which in Step Functions means the callback pattern. You append .waitForTaskToken to the resource ARN, pass $$.Task.Token into the payload, and the execution parks until something calls SendTaskSuccess or SendTaskFailure with that token.

The trap is that a callback task with no timeout waits until the execution itself hits its quota, and Standard workflow executions can run for up to a year. A reviewer who leaves, a review UI that drops the token, a queue consumer that crashes after reading the message and before writing it to the review table: all of these produce an execution that is neither failed nor finished. It just sits there. Nobody alerts on it because nothing broke.

"AwaitAdjusterDecision": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "enqueue-review-task",
    "Payload": {
      "claimId.$": "$.claimId",
      "taskToken.$": "$$.Task.Token"
    }
  },
  "TimeoutSeconds": 259200,
  "HeartbeatSeconds": 3600,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "EscalateStaleReview"
  }],
  "Next": "ApplyDecision"
}

TimeoutSeconds is the maximum total lifetime of the task regardless of heartbeats. HeartbeatSeconds is the maximum gap between SendTaskHeartbeat calls, so a review app that periodically confirms the item is still in someone’s queue will fail fast when that app dies, rather than at the outer limit. AWS’s own guidance is to set the heartbeat below the task timeout for exactly this reason: a heartbeat failure tells you the worker died, a timeout tells you the work took too long, and those are different incidents. Catch States.Timeout and route to a real state. An unhandled timeout is just a differently-shaped silence.

One constraint worth knowing before you design around it: the callback pattern requires Standard workflows. Express workflows support request-response integrations only, so no .waitForTaskToken and no .sync. If you split your pipeline into a fast Express path and a Standard review path, the boundary between them is where the token has to live.

Failure family four: the human review service you can no longer sign up for

This one catches people copying a reference architecture, and it’s the reason to read publication dates on IDP blog posts.

Amazon Augmented AI, known as A2I, was the managed answer to human-in-the-loop review. It plugged directly into Textract’s AnalyzeDocument, watched confidence conditions, and spun up review tasks with a worker UI for you. It appears in a great many architecture diagrams for claims and lending workflows.

Per the AWS documentation, SageMaker A2I is no longer open to new customers. Existing customers can keep using it, and AWS continues security and availability work, but no new features are planned. If you’re standing up a new account today, that diagram does not deploy.

Be fair about what that costs you, because A2I genuinely removed real work: the task assignment logic, the worker UI, the private workforce plumbing through Cognito, result consolidation. Rebuilding it means owning all of that. What you get back is that the review queue becomes yours, which in practice means you can put claim-specific context on the screen instead of a generic key-value editor. For adjusters that difference is not cosmetic.

A minimal replacement is not exotic:

• A DynamoDB table of review items, each holding the claim identifier, the extracted values, the bounding boxes, and the Step Functions task token.
• A small web app for reviewers that renders the page image with the boxes overlaid, so a reviewer confirms placement rather than retyping values.
• An API that writes the corrected values and calls SendTaskSuccess with the stored token.
• Authentication in front of it. Amazon Cognito if you want to stay inside AWS, or an identity-aware proxy such as Cloudflare Access if your reviewers are external adjusters you’d rather not create AWS identities for.
• A sweeper that finds review items older than your heartbeat window and escalates them.

If the review app is a small internal tool with no data residency requirement of its own, it doesn’t have to live in the same account or even the same provider. A modest VPS from a host like Contabo or InterServer running behind a Cloudflare Tunnel is a legitimate answer for a reviewer console that talks to AWS over scoped API credentials. Just be honest about what crosses that boundary, which brings us to the next family.

Failure family five: claim data in places nobody decided to put it

Claims documents carry protected health information, financial identifiers, and often photographs of people and property. The pipeline you drew has three or four places that data lives. The pipeline you deployed has a dozen.

The ones that get missed:

• Lambda logs. One print of an event payload during a debugging session, and CloudWatch Logs is now a claims repository with a different retention policy and a different access model.
• Dead letter queues. A DLQ holds the full failed message. If that message carries extracted values, your DLQ is regulated data, and it is usually the least governed thing in the account.
• Step Functions execution history. State input and output are visible in the console and the history API. Passing extracted fields between states puts them there.
• Intermediate extraction output. The bucket you pointed OutputConfig at holds raw OCR of the whole packet, often with a lifecycle policy nobody wrote.
• Model invocation logging. Bedrock can log inputs and outputs to S3 or CloudWatch. Useful for debugging, and another copy of everything.

The pattern that keeps this manageable is passing pointers, not payloads. States carry an S3 key and a claim identifier; the values themselves stay in one encrypted bucket with one lifecycle policy and one access policy. It makes debugging marginally more annoying and it makes the data map fit on a page.

On regulated workloads, check the current AWS HIPAA-eligible services list and your executed BAA for every service in the path, in the specific region you’re deploying to. Eligibility is per service and it changes. Textract has long been used for claims workflows on that basis, and Bedrock is listed as HIPAA eligible, but “I read a blog post” is not a control. Pull the list yourself before PHI touches anything.

Two smaller things worth deciding early. Reviewers working from home should reach the console over something better than the open internet, whether that’s a corporate tunnel, a business VPN account from a provider like NordVPN or Surfshark, or an identity-aware proxy. And if reviewers ever download claim documents locally, agree what happens to those files afterward, because a deleted file is not an erased file. Tools such as O&O SafeErase exist for exactly that gap on Windows endpoints.

Failure family six: paying twice for the same page

Document AI services bill per page. That single fact reshapes how you think about retries, because in most pipelines a retry is free and here it isn’t.

The expensive patterns are all shaped the same way. A poison document fails a downstream parser, gets requeued, and is re-extracted on every attempt. A batch job re-runs over an entire prefix instead of a delta. A misconfigured S3 event delivers twice. An operator reprocesses a day’s intake to fix a mapping bug in the transform stage, when the extraction stage was fine all along.

The structural fix is separating extraction from interpretation. Extract once, write the raw result to S3 keyed by a content hash of the document, and let every downstream stage read from that. When the mapping bug shows up, you re-run interpretation over stored output and pay nothing. Combined with ClientRequestToken, most accidental double-charges disappear.

Also route the packet before you extract it. Forty pages of accident photographs do not need forms and tables analysis. Classification is cheaper than extraction, and page-level routing is often the single largest lever on the bill.

For attributing that spend, cost allocation tags on the buckets and functions give you the AWS-native view, and platforms like Vantage or CloudZero are worth a look if you need per-claim or per-client unit costs rather than per-service totals. Whatever you use, the metric that matters is cost per claim processed, split by straight-through versus reviewed. Those two numbers tell you whether the automation is earning its keep.

Troubleshooting an insurance claims processing pipeline on AWS

Symptoms you’ll actually see, and where to look first.

• Executions succeed but downstream amounts are wrong. Check whether required fields are present, not just confident. Diff the schema against the response keys for a sample of recent claims. This is failure family one until proven otherwise.
• Executions stuck in Running for days. A callback task with no timeout. List running executions ordered by start time and look for the state name of your review task.
• Throttling on the extraction stage under load. If you’re polling Get operations, move to OutputConfig and SNS notification and stop polling. If you’re already there, check the start-operation limits rather than assuming the whole service is slow.
• The same claim appearing twice. Look for a missing or per-invocation ClientRequestToken, and check whether your S3 event handler is idempotent. Delivery is at-least-once.
• Extraction quality dropped for one document type. Usually the form changed, not the model. Compare bounding boxes for the affected field against an older sample. If the box moved, that’s a layout change, and query aliases or a blueprint update is the fix.
• Review queue growing faster than reviewers clear it. Break the routing reasons apart. If most items are low confidence on one field, that’s an extraction problem wearing a staffing problem’s clothes.
• SNS notification arrives, handler can’t find the results. Confirm the notification role has permission to publish and the handler is reading the S3 prefix rather than calling Get with an expired job identifier.

Common mistakes

• Gating only on confidence, so a missing field is indistinguishable from a clean extraction.
• Defaulting absent values to zero or null in the transform layer instead of raising a routing decision.
• Copying an architecture diagram that includes A2I into a new AWS account.
• Callback tasks with no TimeoutSeconds and no heartbeat.
• Passing extracted claim values through Step Functions state rather than passing an S3 pointer.
• Treating a single global confidence threshold as adequate for every field. A name and a dollar amount do not carry the same downstream risk.
• Running forms and tables analysis over every page of a packet including the photographs.
• No metric for straight-through rate, so nobody notices when it quietly drops.

Best practices worth the effort

• Declare the schema, then validate presence, then confidence, then invariants. Four gates, four distinct rejection reasons, four things a reviewer can act on.
• Set per-field thresholds by consequence. Get the payable amount wrong and money moves. Get a street suffix wrong and a letter is slightly odd.
• Keep a labeled regression set. A few dozen real packets with known-correct values, run on every blueprint or query change. Bedrock Data Automation can use ground-truth examples to refine blueprint instructions, which only works if you maintain the ground truth.
• Store bounding boxes alongside values. They make review faster and they turn “quality dropped” from a guess into a comparison.
• Alarm on rates, not just errors. Straight-through rate, review-queue age, and cost per claim. A pipeline that stops approving anything is broken even though nothing threw.
• Make every stage idempotent on a content hash. Reprocessing is normal. It should be safe and cheap.
• Instrument the pipeline like a pipeline. CloudWatch covers the AWS surface; if you’re consolidating with on-premises claims systems, a platform like Grafana Cloud gives you one place to correlate both sides.

Frequently asked questions

Should I use Amazon Textract or Bedrock Data Automation for claims extraction?

Textract is the sharper tool when your documents are standardized forms and you want deterministic OCR with forms, tables and targeted queries. Bedrock Data Automation is stronger on mixed packets, because it splits along logical document boundaries, classifies each part, and applies a blueprint per document type, with confidence scores and visual grounding on the output. Claims intake is usually mixed packets, which tilts toward Data Automation, but the honest answer is to run both against a sample of your real documents. The evaluation costs a day and it decides your architecture.

What replaces Amazon A2I for human review?

For new AWS accounts, a custom review path: a queue or table of review items, a reviewer UI, and the Step Functions callback pattern to resume the workflow. It’s more code than A2I but not a large amount, and it gives you a review screen designed around claims rather than around generic key-value pairs. Existing A2I customers can continue as they are, though building on a service with no planned features is a decision to make deliberately rather than by default.

What straight-through processing rate should I expect?

Anyone quoting you a number without seeing your documents is guessing. It depends almost entirely on document quality and how many fields you require. What’s reliable is the method: measure your current rate, split failures by reason, and fix the largest reason. Requiring one rarely-present field can dominate everything else, and that’s a policy decision as much as an engineering one.

How do I keep an insurance claims processing pipeline on AWS HIPAA-aligned?

Start from the AWS HIPAA-eligible services list and an executed BAA, and confirm eligibility for each service in your specific region. Then do the unglamorous work: customer-managed KMS keys, no PHI in logs or state payloads, scoped IAM roles per stage, VPC endpoints where the service supports them, retention policies on every bucket and queue including dead letter queues, and CloudTrail configured so you can answer who accessed which claim. Eligibility is permission to build; the controls are yours.

Can I run the whole pipeline with Step Functions Express workflows?

Not the part that waits for a human. Express workflows support request-response integrations only, so the callback pattern requires Standard. A common split is Express for the high-volume deterministic stages and Standard for anything holding a task token, with the two connected by an event or a queue.

How do I stop duplicate claims from duplicate events?

Treat every trigger as at-least-once. Derive an idempotency key from the document itself, usually a content hash plus a version marker, pass it as ClientRequestToken to the extraction call, and use it as the conditional write key when you create the claim record. Then a duplicate event is a no-op rather than a second claim.

Is it worth starting from the AWS GenAI IDP Accelerator?

As a reference for structure and as a way to get a working pipeline in front of stakeholders quickly, yes. As a production system you inherit wholesale, be careful: you’re adopting someone else’s opinions about classification, review and storage, and you’ll be reading that code anyway the first time something behaves oddly. Read it, borrow the patterns, own what you deploy.

The one thing to take away

An insurance claims processing pipeline on AWS is not hard to build. Textract, Bedrock Data Automation, Step Functions and S3 will get you a working pipeline in a couple of weeks. What’s hard is making it fail in ways you can see.

Every expensive failure in this space shares one shape: the pipeline had no opinion about what it did not receive. A field that didn’t come back scored nothing, a callback that never fired errored nothing, a duplicate event failed nothing. If you take one design rule from this, take that one. Declare what a complete claim looks like, check for its absence explicitly, and route anything incomplete to a human with a reason attached.

Green executions are not evidence. Reconciled numbers are.

Need help with your claims pipeline?

I work with teams building document-heavy workflows on AWS, and claims pipelines are one of the places where a small amount of design care prevents a large amount of reconciliation work. Things I can help with:

• Reviewing an existing extraction pipeline for silent-failure paths, particularly missing-field handling and default values in the transform layer
• Designing the routing logic: schema gates, per-field thresholds, cross-field invariants, and the escalation rules that sit behind them
• Building a human review path with the Step Functions callback pattern, including timeouts, heartbeats and a sweeper for stale tasks
• Running a structured evaluation of Amazon Textract against Bedrock Data Automation on your actual documents, with a labeled regression set you keep afterward
• Tracing where claim data actually lands across logs, queues, execution history and intermediate buckets, then shrinking that footprint
• Instrumenting straight-through rate, review-queue age and cost per claim so regressions surface before finance finds them

If you’d like a second opinion, send me a state machine definition, a sample extraction response with the values redacted, or the routing code that decides what goes to review. That’s usually enough to spot the gap.

The post Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly appeared first on John Nessime.

Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify’s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.

That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.

That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn’t there.

This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.

What the pipe actually looks like

Four moving parts, and only two of them live in your account.

1. A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.
2. Shopify creates a partner event source inside your AWS account, in the region you nominated.
3. You associate that source with a partner event bus. This is the step everyone forgets.
4. Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.

The ARN trips people up more than anything else in the setup. Shopify wants the event source ARN, not the event bus ARN. They look similar and only one of them works:

# Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/<id>/<source-name>

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/<id>/<source-name>

Associating the source is a single call, and both the name and the source name are the same string:

# Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --event-source-name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge’s semantics, not the store’s.

Failure family one: events that never existed

This is the one from the opening, and it is the most expensive because it is completely silent on both sides.

Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.

The same class of hole opens up in two other ways:

• Region mismatch. The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.
• Environment drift. A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.

The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls describe-event-source and alarms if State is anything other than ACTIVE costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.

The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on MatchedEvents hitting zero, not just on errors.

Failure family two: events that arrive twice, or backwards

EventBridge is at-least-once. Shopify’s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.

The duplicate is the obvious one. The same orders/create arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts X-Shopify-Webhook-Id into detail.metadata, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.

The out-of-order case is the one that costs you money quietly. An orders/updated carrying a cancelled status arrives before the orders/updated carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.

The envelope carries what you need for this too. detail.metadata includes X-Shopify-Triggered-At, and the resource in detail.payload carries its own updated_at. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.

The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.

One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.

Failure family three: rules that match nothing

Every Shopify event, regardless of topic, arrives with the same detail-type. That single fact invalidates the routing instinct most people bring from AWS service events.

{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/<id>/<source-name>",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}

Two things to take from that envelope. The resource body is nested under detail.payload, not at the top of detail, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in detail.metadata, which is where all your routing has to happen.

// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}

Do not deploy a pattern you have not tested against a real envelope. test-event-pattern answers in a second and saves an afternoon:

aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json

One rule or thirty?

There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.

What you give up is per-topic visibility. MatchedEvents, FailedInvocations and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.

The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.

Failure family four: events that arrive and die at the target

By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.

First, a whole class of errors gets no retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.

Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.

aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/<id>/<source-name>" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'

Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.

The DLQ permission trap

This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through PutTargets — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.

{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}

The metric that catches this is InvocationsFailedToBeSentToDlq. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.

Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.

Failure family five: the bill

EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.

This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.

Three levers, roughly in order of how much they return:

• Trim at the subscription. Shopify’s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.
• Subscribe to fewer topics. Broad topics like orders/updated fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.
• Archive selectively, and set retention. Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.

One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.

The layer nobody builds until they need it

Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that reached the bus. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.

For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.

• Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.
• Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.
• Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.
• Emit the drift as a metric, not just a log line. “Orders in Shopify but not in our store, last hour” is a graph worth putting on a dashboard, and it should normally read zero.

The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.

Troubleshooting by symptom

Work these in order. Each one is cheap and rules out a whole branch.

Nothing is arriving at all

1. Run describe-event-source. If State is not ACTIVE, stop here. Everything published so far is gone and you need the reconciliation path.
2. Confirm the region of the bus matches the region in the source ARN.
3. List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.
4. Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.
5. Check MatchedEvents on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.

Some topics arrive, order or customer topics do not

This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.

The rule matches but the target does nothing

Compare MatchedEvents against SuccessfulInvocationAttempts on the rule. A gap sends you to FailedInvocations and to the DLQ. Check the target’s resource policy, and check InvocationsFailedToBeSentToDlq before you trust that the DLQ is catching anything.

Events arrive, but late

Look at ThrottledRules and at IngestionToInvocationSuccessLatency. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.

Common mistakes

• Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.
• Registering the event bus ARN instead of the event source ARN, then debugging Shopify’s rejection for an hour.
• Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under detail.payload.
• Routing on detail-type. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.
• Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.
• Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.
• Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.
• Skipping reconciliation because the pipeline “works”. It works right up until it doesn’t, and that is precisely when you need the other path.

Best practices for streaming Shopify events into AWS

• Alarm on the event source state and on MatchedEvents reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.
• Dedupe on X-Shopify-Webhook-Id and version-check on X-Shopify-Triggered-At. Two mechanisms, two problems.
• Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.
• Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.
• Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.
• Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.
• Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.
• Build reconciliation before your first peak trading period, not after your first missing-order ticket.

Frequently asked questions

Do I still need to verify the HMAC signature on the EventBridge path?

No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.

Should I use EventBridge or plain HTTPS webhooks?

HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.

Can I use one partner event source for multiple stores?

Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by X-Shopify-Shop-Domain in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.

Why do I get duplicate order events even though nothing failed?

Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several orders/updated events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.

What happens to events published while my consumer is broken?

They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.

Can I archive and replay Shopify events?

Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.

Does this work the same way for BigCommerce or commercetools?

The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.

The one thing worth remembering

Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.

The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.

Need a hand with your event pipeline?

Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:

• Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.
• Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.
• Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.
• Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.
• Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.
• Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.

If you have a rule pattern that isn’t matching, a DLQ that’s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I’ll tell you what I’d look at first.

The post Streaming Shopify Events into AWS Without Losing Orders appeared first on John Nessime.