That is the failure mode that defines this problem. An insurance claims processing pipeline on AWS almost never falls over in the way you designed it to fall over. It succeeds. It emits well-formed JSON. It hands a number to a payment system, and the number is wrong, and nothing in the pipeline had any reason to think otherwise.

This post is organized by failure family rather than by service. I’ll walk through the six ways these pipelines go quietly wrong, what each one costs, and what the fix actually looks like in Textract, Bedrock, Step Functions and S3. There’s a troubleshooting section, the mistakes I see repeated, and an FAQ at the end.

The shape most claims pipelines end up with

Before the failure families make sense, the skeleton. Almost every serverless claims pipeline lands on roughly the same set of stages, whatever the vendor deck calls them:

1. Intake. A document lands in S3 from a portal upload, an SFTP drop, or a mail scanning vendor. An S3 event or EventBridge rule starts an execution.
2. Classification. Work out what the packet actually contains. A first notice of loss, a CMS-style claim form, a police report, an itemized bill, forty pages of photographs.
3. Extraction. Pull the fields you need. Amazon Textract for OCR, forms and tables, or Amazon Bedrock Data Automation with a blueprint that names the fields directly.
4. Validation. Check the extracted values against business rules, policy data, and each other.
5. Routing. Straight-through processing, human review, or rejection with a reason.
6. Persistence and audit. The claim record, the extraction artifacts, and enough evidence to explain a decision months later.

Nothing controversial there. AWS publishes an open-source GenAI IDP Accelerator that implements exactly this shape, with a Bedrock Data Automation mode and a Textract-plus-foundation-model pipeline mode, and it’s a reasonable place to start reading. The interesting part is not the boxes. It’s what happens between them.

Failure family one: the field that was never there

This is the one that pays out the wrong number, and it is worth more attention than everything else in this post combined.

Every extraction service gives you confidence scores. So the obvious design is a gate: if every field scores above some threshold, approve automatically; if anything falls below, send it to a human. That gate is sound reasoning applied to the wrong population.

A confidence score only exists for a value that came back. When the extractor doesn’t find a field at all, there is no low score to catch, because there’s nothing to score. The gate iterates over four returned fields, finds all four above threshold, and reports a clean pass. The fifth field, the one that determines coordination of benefits or the deductible offset, is simply absent from the response. Downstream code treats absent as zero, or as null, or as “not applicable,” and the claim goes through.

The fix is structural, not statistical. Validate presence against a schema before you validate confidence, and treat the two as separate gates with separate outcomes:

# Two gates, not one. Presence first, then confidence.
# 'extracted' is the flattened field map from Textract Queries
# or a Bedrock Data Automation blueprint result.

REQUIRED = {
    "claim_number",
    "date_of_service",
    "billed_amount",
    "member_id",
    "secondary_payer_indicator",
}

def gate(extracted, scores, threshold=0.95):
    missing = REQUIRED - set(extracted)
    if missing:
        # Never silently default. This is a routing decision.
        return "HUMAN_REVIEW", {"reason": "missing_fields",
                                "fields": sorted(missing)}

    weak = [f for f in REQUIRED if scores.get(f, 0.0) < threshold]
    if weak:
        return "HUMAN_REVIEW", {"reason": "low_confidence",
                                "fields": sorted(weak)}

    return "STRAIGHT_THROUGH", {}

Two details matter here. The set difference is computed against a declared schema, not against whatever keys happen to be in the response, so an absent field becomes a first-class routing reason. And scores.get(f, 0.0) defaults to zero rather than to a passing value, so a field that arrives without a score fails closed.

If you’re on Textract Queries, there’s a second reason to be explicit: Queries let you attach an alias to each question, which means your schema keys are yours rather than whatever label happened to be printed on the form. That’s the difference between “the field is missing” and “the field moved and we didn’t notice.”

Failure family two: confidence scores that answer a different question

Assume you’ve fixed presence. The next trap is what the confidence number is measuring.

OCR confidence is a statement about characters. It says the model is highly sure those pixels read 1,240.00. It is not a statement that 1,240.00 is the billed amount rather than the allowed amount from the box directly above it, or the prior balance from a remittance summary that happened to be stapled into the same packet. Read it as a legibility score, because that’s closer to what it is.

Bedrock Data Automation narrows this gap: blueprints define fields semantically, confidence scores come with bounding boxes, and the visual grounding lets you point at the region a value came from. Textract Queries narrow it too, by asking a question rather than harvesting a label. Neither eliminates the problem, because a high-confidence read of the wrong region still scores high.

What actually catches this is cross-field invariants. They cost almost nothing and they fail for reasons a human can read:

• Arithmetic. Line items sum to the claimed total. If they don’t, one of the two is wrong and you don’t yet know which.
• Temporal. Date of service falls inside the policy period and before the date of submission. A service date after the submission date is a parsing error nine times out of ten.
• Referential. The member or policy identifier resolves against your system of record. An identifier that matches the format but not a real record is a strong signal you read the wrong box.
• Range. Amounts within a plausible band for the claim type. Not a fraud model, just a tripwire for a decimal point that moved.
• Page provenance. Fields that must come from the same page or the same document within the packet. Bounding box data makes this checkable rather than assumed.

An invariant failure is more useful than a low score, because it names a contradiction. “Line items sum to 1,180 but the claimed total reads 1,240” is something a reviewer resolves in seconds. “Confidence 0.91” is something a reviewer stares at.

Failure family three: the claim that stops halfway

Claims documents are multi-page packets, so you’ll be using Textract’s asynchronous operations. That means jobs, notifications, and a whole class of orchestration bugs that only show up under load or after a weekend.

The asynchronous pattern is: call StartDocumentAnalysis, get a JobId back, and let Textract publish completion to an SNS topic you nominate. A request looks like this:

{
  "DocumentLocation": {
    "S3Object": { "Bucket": "claims-intake", "Name": "packets/abc123.pdf" }
  },
  "FeatureTypes": ["FORMS", "TABLES"],
  "ClientRequestToken": "abc123-v1",
  "JobTag": "fnol-packet",
  "NotificationChannel": {
    "SNSTopicArn": "arn:aws:sns:REGION:ACCOUNT:textract-complete",
    "RoleArn": "arn:aws:iam::ACCOUNT:role/TextractPublishRole"
  },
  "OutputConfig": {
    "S3Bucket": "claims-extraction",
    "S3Prefix": "raw/"
  },
  "KMSKeyId": "alias/claims-cmk"
}

Three of those parameters are doing load-bearing work that is easy to skip.

ClientRequestToken is the idempotency token. Reuse the same token and you get the same JobId back instead of a second job. Derive it from the document, not from the invocation, and a Lambda retry or a duplicated S3 event stops turning into a duplicate charge and a duplicate claim record.

OutputConfig writes results into a bucket you control. Without it, results stay internal to Textract and the only way to read them is the Get operations, which have their own throttling limits. Under concurrency those limits become the bottleneck: you end up polling more jobs than you’re allowed to poll, backing off, and watching end-to-end latency climb for reasons that have nothing to do with the documents. Writing to S3 sidesteps the whole path.

JobTag shows up in the completion notification. In a mixed pipeline where the same topic carries first notice of loss packets, itemized bills and ID documents, that tag is what lets the notification handler route without a lookup.

One expiry to plan around: a Textract JobId is only valid for seven days. If your retry story is “requeue it and someone will look on Monday,” a bad weekend turns recoverable failures into full reprocessing. Persist the S3 output location, not the job identifier.

Callbacks that never come back

Human review means pausing a workflow for hours or days, which in Step Functions means the callback pattern. You append .waitForTaskToken to the resource ARN, pass $$.Task.Token into the payload, and the execution parks until something calls SendTaskSuccess or SendTaskFailure with that token.

The trap is that a callback task with no timeout waits until the execution itself hits its quota, and Standard workflow executions can run for up to a year. A reviewer who leaves, a review UI that drops the token, a queue consumer that crashes after reading the message and before writing it to the review table: all of these produce an execution that is neither failed nor finished. It just sits there. Nobody alerts on it because nothing broke.

"AwaitAdjusterDecision": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "enqueue-review-task",
    "Payload": {
      "claimId.$": "$.claimId",
      "taskToken.$": "$$.Task.Token"
    }
  },
  "TimeoutSeconds": 259200,
  "HeartbeatSeconds": 3600,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "EscalateStaleReview"
  }],
  "Next": "ApplyDecision"
}

TimeoutSeconds is the maximum total lifetime of the task regardless of heartbeats. HeartbeatSeconds is the maximum gap between SendTaskHeartbeat calls, so a review app that periodically confirms the item is still in someone’s queue will fail fast when that app dies, rather than at the outer limit. AWS’s own guidance is to set the heartbeat below the task timeout for exactly this reason: a heartbeat failure tells you the worker died, a timeout tells you the work took too long, and those are different incidents. Catch States.Timeout and route to a real state. An unhandled timeout is just a differently-shaped silence.

One constraint worth knowing before you design around it: the callback pattern requires Standard workflows. Express workflows support request-response integrations only, so no .waitForTaskToken and no .sync. If you split your pipeline into a fast Express path and a Standard review path, the boundary between them is where the token has to live.

Failure family four: the human review service you can no longer sign up for

This one catches people copying a reference architecture, and it’s the reason to read publication dates on IDP blog posts.

Amazon Augmented AI, known as A2I, was the managed answer to human-in-the-loop review. It plugged directly into Textract’s AnalyzeDocument, watched confidence conditions, and spun up review tasks with a worker UI for you. It appears in a great many architecture diagrams for claims and lending workflows.

Per the AWS documentation, SageMaker A2I is no longer open to new customers. Existing customers can keep using it, and AWS continues security and availability work, but no new features are planned. If you’re standing up a new account today, that diagram does not deploy.

Be fair about what that costs you, because A2I genuinely removed real work: the task assignment logic, the worker UI, the private workforce plumbing through Cognito, result consolidation. Rebuilding it means owning all of that. What you get back is that the review queue becomes yours, which in practice means you can put claim-specific context on the screen instead of a generic key-value editor. For adjusters that difference is not cosmetic.

A minimal replacement is not exotic:

• A DynamoDB table of review items, each holding the claim identifier, the extracted values, the bounding boxes, and the Step Functions task token.
• A small web app for reviewers that renders the page image with the boxes overlaid, so a reviewer confirms placement rather than retyping values.
• An API that writes the corrected values and calls SendTaskSuccess with the stored token.
• Authentication in front of it. Amazon Cognito if you want to stay inside AWS, or an identity-aware proxy such as Cloudflare Access if your reviewers are external adjusters you’d rather not create AWS identities for.
• A sweeper that finds review items older than your heartbeat window and escalates them.

If the review app is a small internal tool with no data residency requirement of its own, it doesn’t have to live in the same account or even the same provider. A modest VPS from a host like Contabo or InterServer running behind a Cloudflare Tunnel is a legitimate answer for a reviewer console that talks to AWS over scoped API credentials. Just be honest about what crosses that boundary, which brings us to the next family.

Failure family five: claim data in places nobody decided to put it

Claims documents carry protected health information, financial identifiers, and often photographs of people and property. The pipeline you drew has three or four places that data lives. The pipeline you deployed has a dozen.

The ones that get missed:

• Lambda logs. One print of an event payload during a debugging session, and CloudWatch Logs is now a claims repository with a different retention policy and a different access model.
• Dead letter queues. A DLQ holds the full failed message. If that message carries extracted values, your DLQ is regulated data, and it is usually the least governed thing in the account.
• Step Functions execution history. State input and output are visible in the console and the history API. Passing extracted fields between states puts them there.
• Intermediate extraction output. The bucket you pointed OutputConfig at holds raw OCR of the whole packet, often with a lifecycle policy nobody wrote.
• Model invocation logging. Bedrock can log inputs and outputs to S3 or CloudWatch. Useful for debugging, and another copy of everything.

The pattern that keeps this manageable is passing pointers, not payloads. States carry an S3 key and a claim identifier; the values themselves stay in one encrypted bucket with one lifecycle policy and one access policy. It makes debugging marginally more annoying and it makes the data map fit on a page.

On regulated workloads, check the current AWS HIPAA-eligible services list and your executed BAA for every service in the path, in the specific region you’re deploying to. Eligibility is per service and it changes. Textract has long been used for claims workflows on that basis, and Bedrock is listed as HIPAA eligible, but “I read a blog post” is not a control. Pull the list yourself before PHI touches anything.

Two smaller things worth deciding early. Reviewers working from home should reach the console over something better than the open internet, whether that’s a corporate tunnel, a business VPN account from a provider like NordVPN or Surfshark, or an identity-aware proxy. And if reviewers ever download claim documents locally, agree what happens to those files afterward, because a deleted file is not an erased file. Tools such as O&O SafeErase exist for exactly that gap on Windows endpoints.

Failure family six: paying twice for the same page

Document AI services bill per page. That single fact reshapes how you think about retries, because in most pipelines a retry is free and here it isn’t.

The expensive patterns are all shaped the same way. A poison document fails a downstream parser, gets requeued, and is re-extracted on every attempt. A batch job re-runs over an entire prefix instead of a delta. A misconfigured S3 event delivers twice. An operator reprocesses a day’s intake to fix a mapping bug in the transform stage, when the extraction stage was fine all along.

The structural fix is separating extraction from interpretation. Extract once, write the raw result to S3 keyed by a content hash of the document, and let every downstream stage read from that. When the mapping bug shows up, you re-run interpretation over stored output and pay nothing. Combined with ClientRequestToken, most accidental double-charges disappear.

Also route the packet before you extract it. Forty pages of accident photographs do not need forms and tables analysis. Classification is cheaper than extraction, and page-level routing is often the single largest lever on the bill.

For attributing that spend, cost allocation tags on the buckets and functions give you the AWS-native view, and platforms like Vantage or CloudZero are worth a look if you need per-claim or per-client unit costs rather than per-service totals. Whatever you use, the metric that matters is cost per claim processed, split by straight-through versus reviewed. Those two numbers tell you whether the automation is earning its keep.

Troubleshooting an insurance claims processing pipeline on AWS

Symptoms you’ll actually see, and where to look first.

• Executions succeed but downstream amounts are wrong. Check whether required fields are present, not just confident. Diff the schema against the response keys for a sample of recent claims. This is failure family one until proven otherwise.
• Executions stuck in Running for days. A callback task with no timeout. List running executions ordered by start time and look for the state name of your review task.
• Throttling on the extraction stage under load. If you’re polling Get operations, move to OutputConfig and SNS notification and stop polling. If you’re already there, check the start-operation limits rather than assuming the whole service is slow.
• The same claim appearing twice. Look for a missing or per-invocation ClientRequestToken, and check whether your S3 event handler is idempotent. Delivery is at-least-once.
• Extraction quality dropped for one document type. Usually the form changed, not the model. Compare bounding boxes for the affected field against an older sample. If the box moved, that’s a layout change, and query aliases or a blueprint update is the fix.
• Review queue growing faster than reviewers clear it. Break the routing reasons apart. If most items are low confidence on one field, that’s an extraction problem wearing a staffing problem’s clothes.
• SNS notification arrives, handler can’t find the results. Confirm the notification role has permission to publish and the handler is reading the S3 prefix rather than calling Get with an expired job identifier.

Common mistakes

• Gating only on confidence, so a missing field is indistinguishable from a clean extraction.
• Defaulting absent values to zero or null in the transform layer instead of raising a routing decision.
• Copying an architecture diagram that includes A2I into a new AWS account.
• Callback tasks with no TimeoutSeconds and no heartbeat.
• Passing extracted claim values through Step Functions state rather than passing an S3 pointer.
• Treating a single global confidence threshold as adequate for every field. A name and a dollar amount do not carry the same downstream risk.
• Running forms and tables analysis over every page of a packet including the photographs.
• No metric for straight-through rate, so nobody notices when it quietly drops.

Best practices worth the effort

• Declare the schema, then validate presence, then confidence, then invariants. Four gates, four distinct rejection reasons, four things a reviewer can act on.
• Set per-field thresholds by consequence. Get the payable amount wrong and money moves. Get a street suffix wrong and a letter is slightly odd.
• Keep a labeled regression set. A few dozen real packets with known-correct values, run on every blueprint or query change. Bedrock Data Automation can use ground-truth examples to refine blueprint instructions, which only works if you maintain the ground truth.
• Store bounding boxes alongside values. They make review faster and they turn “quality dropped” from a guess into a comparison.
• Alarm on rates, not just errors. Straight-through rate, review-queue age, and cost per claim. A pipeline that stops approving anything is broken even though nothing threw.
• Make every stage idempotent on a content hash. Reprocessing is normal. It should be safe and cheap.
• Instrument the pipeline like a pipeline. CloudWatch covers the AWS surface; if you’re consolidating with on-premises claims systems, a platform like Grafana Cloud gives you one place to correlate both sides.

Frequently asked questions

Should I use Amazon Textract or Bedrock Data Automation for claims extraction?

Textract is the sharper tool when your documents are standardized forms and you want deterministic OCR with forms, tables and targeted queries. Bedrock Data Automation is stronger on mixed packets, because it splits along logical document boundaries, classifies each part, and applies a blueprint per document type, with confidence scores and visual grounding on the output. Claims intake is usually mixed packets, which tilts toward Data Automation, but the honest answer is to run both against a sample of your real documents. The evaluation costs a day and it decides your architecture.

What replaces Amazon A2I for human review?

For new AWS accounts, a custom review path: a queue or table of review items, a reviewer UI, and the Step Functions callback pattern to resume the workflow. It’s more code than A2I but not a large amount, and it gives you a review screen designed around claims rather than around generic key-value pairs. Existing A2I customers can continue as they are, though building on a service with no planned features is a decision to make deliberately rather than by default.

What straight-through processing rate should I expect?

Anyone quoting you a number without seeing your documents is guessing. It depends almost entirely on document quality and how many fields you require. What’s reliable is the method: measure your current rate, split failures by reason, and fix the largest reason. Requiring one rarely-present field can dominate everything else, and that’s a policy decision as much as an engineering one.

How do I keep an insurance claims processing pipeline on AWS HIPAA-aligned?

Start from the AWS HIPAA-eligible services list and an executed BAA, and confirm eligibility for each service in your specific region. Then do the unglamorous work: customer-managed KMS keys, no PHI in logs or state payloads, scoped IAM roles per stage, VPC endpoints where the service supports them, retention policies on every bucket and queue including dead letter queues, and CloudTrail configured so you can answer who accessed which claim. Eligibility is permission to build; the controls are yours.

Can I run the whole pipeline with Step Functions Express workflows?

Not the part that waits for a human. Express workflows support request-response integrations only, so the callback pattern requires Standard. A common split is Express for the high-volume deterministic stages and Standard for anything holding a task token, with the two connected by an event or a queue.

How do I stop duplicate claims from duplicate events?

Treat every trigger as at-least-once. Derive an idempotency key from the document itself, usually a content hash plus a version marker, pass it as ClientRequestToken to the extraction call, and use it as the conditional write key when you create the claim record. Then a duplicate event is a no-op rather than a second claim.

Is it worth starting from the AWS GenAI IDP Accelerator?

As a reference for structure and as a way to get a working pipeline in front of stakeholders quickly, yes. As a production system you inherit wholesale, be careful: you’re adopting someone else’s opinions about classification, review and storage, and you’ll be reading that code anyway the first time something behaves oddly. Read it, borrow the patterns, own what you deploy.

The one thing to take away

An insurance claims processing pipeline on AWS is not hard to build. Textract, Bedrock Data Automation, Step Functions and S3 will get you a working pipeline in a couple of weeks. What’s hard is making it fail in ways you can see.

Every expensive failure in this space shares one shape: the pipeline had no opinion about what it did not receive. A field that didn’t come back scored nothing, a callback that never fired errored nothing, a duplicate event failed nothing. If you take one design rule from this, take that one. Declare what a complete claim looks like, check for its absence explicitly, and route anything incomplete to a human with a reason attached.

Green executions are not evidence. Reconciled numbers are.

Need help with your claims pipeline?

I work with teams building document-heavy workflows on AWS, and claims pipelines are one of the places where a small amount of design care prevents a large amount of reconciliation work. Things I can help with:

• Reviewing an existing extraction pipeline for silent-failure paths, particularly missing-field handling and default values in the transform layer
• Designing the routing logic: schema gates, per-field thresholds, cross-field invariants, and the escalation rules that sit behind them
• Building a human review path with the Step Functions callback pattern, including timeouts, heartbeats and a sweeper for stale tasks
• Running a structured evaluation of Amazon Textract against Bedrock Data Automation on your actual documents, with a labeled regression set you keep afterward
• Tracing where claim data actually lands across logs, queues, execution history and intermediate buckets, then shrinking that footprint
• Instrumenting straight-through rate, review-queue age and cost per claim so regressions surface before finance finds them

If you’d like a second opinion, send me a state machine definition, a sample extraction response with the values redacted, or the routing code that decides what goes to review. That’s usually enough to spot the gap.

The post Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly appeared first on John Nessime.

Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify’s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.

That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.

That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn’t there.

This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.

What the pipe actually looks like

Four moving parts, and only two of them live in your account.

1. A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.
2. Shopify creates a partner event source inside your AWS account, in the region you nominated.
3. You associate that source with a partner event bus. This is the step everyone forgets.
4. Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.

The ARN trips people up more than anything else in the setup. Shopify wants the event source ARN, not the event bus ARN. They look similar and only one of them works:

# Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/<id>/<source-name>

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/<id>/<source-name>

Associating the source is a single call, and both the name and the source name are the same string:

# Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --event-source-name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge’s semantics, not the store’s.

Failure family one: events that never existed

This is the one from the opening, and it is the most expensive because it is completely silent on both sides.

Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.

The same class of hole opens up in two other ways:

• Region mismatch. The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.
• Environment drift. A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.

The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls describe-event-source and alarms if State is anything other than ACTIVE costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.

The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on MatchedEvents hitting zero, not just on errors.

Failure family two: events that arrive twice, or backwards

EventBridge is at-least-once. Shopify’s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.

The duplicate is the obvious one. The same orders/create arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts X-Shopify-Webhook-Id into detail.metadata, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.

The out-of-order case is the one that costs you money quietly. An orders/updated carrying a cancelled status arrives before the orders/updated carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.

The envelope carries what you need for this too. detail.metadata includes X-Shopify-Triggered-At, and the resource in detail.payload carries its own updated_at. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.

The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.

One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.

Failure family three: rules that match nothing

Every Shopify event, regardless of topic, arrives with the same detail-type. That single fact invalidates the routing instinct most people bring from AWS service events.

{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/<id>/<source-name>",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}

Two things to take from that envelope. The resource body is nested under detail.payload, not at the top of detail, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in detail.metadata, which is where all your routing has to happen.

// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}

Do not deploy a pattern you have not tested against a real envelope. test-event-pattern answers in a second and saves an afternoon:

aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json

One rule or thirty?

There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.

What you give up is per-topic visibility. MatchedEvents, FailedInvocations and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.

The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.

Failure family four: events that arrive and die at the target

By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.

First, a whole class of errors gets no retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.

Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.

aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/<id>/<source-name>" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'

Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.

The DLQ permission trap

This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through PutTargets — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.

{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}

The metric that catches this is InvocationsFailedToBeSentToDlq. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.

Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.

Failure family five: the bill

EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.

This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.

Three levers, roughly in order of how much they return:

• Trim at the subscription. Shopify’s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.
• Subscribe to fewer topics. Broad topics like orders/updated fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.
• Archive selectively, and set retention. Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.

One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.

The layer nobody builds until they need it

Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that reached the bus. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.

For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.

• Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.
• Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.
• Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.
• Emit the drift as a metric, not just a log line. “Orders in Shopify but not in our store, last hour” is a graph worth putting on a dashboard, and it should normally read zero.

The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.

Troubleshooting by symptom

Work these in order. Each one is cheap and rules out a whole branch.

Nothing is arriving at all

1. Run describe-event-source. If State is not ACTIVE, stop here. Everything published so far is gone and you need the reconciliation path.
2. Confirm the region of the bus matches the region in the source ARN.
3. List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.
4. Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.
5. Check MatchedEvents on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.

Some topics arrive, order or customer topics do not

This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.

The rule matches but the target does nothing

Compare MatchedEvents against SuccessfulInvocationAttempts on the rule. A gap sends you to FailedInvocations and to the DLQ. Check the target’s resource policy, and check InvocationsFailedToBeSentToDlq before you trust that the DLQ is catching anything.

Events arrive, but late

Look at ThrottledRules and at IngestionToInvocationSuccessLatency. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.

Common mistakes

• Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.
• Registering the event bus ARN instead of the event source ARN, then debugging Shopify’s rejection for an hour.
• Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under detail.payload.
• Routing on detail-type. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.
• Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.
• Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.
• Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.
• Skipping reconciliation because the pipeline “works”. It works right up until it doesn’t, and that is precisely when you need the other path.

Best practices for streaming Shopify events into AWS

• Alarm on the event source state and on MatchedEvents reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.
• Dedupe on X-Shopify-Webhook-Id and version-check on X-Shopify-Triggered-At. Two mechanisms, two problems.
• Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.
• Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.
• Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.
• Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.
• Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.
• Build reconciliation before your first peak trading period, not after your first missing-order ticket.

Frequently asked questions

Do I still need to verify the HMAC signature on the EventBridge path?

No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.

Should I use EventBridge or plain HTTPS webhooks?

HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.

Can I use one partner event source for multiple stores?

Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by X-Shopify-Shop-Domain in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.

Why do I get duplicate order events even though nothing failed?

Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several orders/updated events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.

What happens to events published while my consumer is broken?

They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.

Can I archive and replay Shopify events?

Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.

Does this work the same way for BigCommerce or commercetools?

The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.

The one thing worth remembering

Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.

The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.

Need a hand with your event pipeline?

Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:

• Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.
• Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.
• Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.
• Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.
• Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.
• Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.

If you have a rule pattern that isn’t matching, a DLQ that’s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I’ll tell you what I’d look at first.

The post Streaming Shopify Events into AWS Without Losing Orders appeared first on John Nessime.