That is the failure mode that actually bites when you build a medical lab data pipeline on AWS. Not the bucket you thought about. The seven places PHI ended up because a payload got logged, an error got queued, or a filename contained a medical record number.

This post walks through the design boundary by boundary: where lab data crosses from one trust zone into another, what leaks at each crossing, and which control closes it. It is aimed at engineers who have been handed an HL7 feed and a compliance checklist and told to make them meet. I am an engineer and not your compliance counsel, so treat the regulatory references here as pointers to read the primary text yourself.

Why laboratory data is a harder pipeline than it looks

Lab data has three properties that most ETL work does not.

First, results get corrected. A preliminary result goes out, the analyzer is recalibrated, and a corrected result follows hours or days later. Your pipeline is not append-only in the way a clickstream is. It has to carry amendment semantics all the way through, and an analytics table that silently keeps the first value is worse than no table at all.

Second, the wire format is old and the transport is older. Most laboratory information systems still emit HL7 version 2 messages, pipe-delimited segments carried over the Minimum Lower Layer Protocol, which is framed bytes over a raw TCP socket. AWS documents MLLP as the common transport standard for HL7 v2 interoperability in its Healthcare Industry Lens. It has no built-in authentication and no built-in encryption. Whatever security it has, you bolt on around it.

Third, the identifiers are load-bearing. Accession numbers, specimen IDs and medical record numbers are how the lab reconciles anything, so PHI is not a column you can drop. It is structural, and it will try to escape into every metadata surface you own.

Boundary one: getting messages out of the lab

The instrument or LIS sits on a network you probably do not control, run by a vendor who may not let you touch the configuration. That constrains the design more than anything on the AWS side.

You get three realistic shapes, and the LIS vendor usually picks for you.

• Live MLLP over a private link. An interface engine in the lab holds the TCP session and forwards over a Site-to-Site VPN or Direct Connect into a private subnet. Lowest latency, highest operational burden, because a dropped socket is a clinical incident and not a retry.
• Batch file drop over SFTP. The LIS writes flat files on a schedule and something picks them up. Far easier to reason about, far easier to audit, and acceptable whenever the downstream use is analytics rather than result delivery.
• An outbound API push. Newer systems will POST to an endpoint you expose. Best case, because you control authentication and you get a response code back, but rarest in practice.

Whichever shape you get, the same rule applies at this boundary: terminate the untrusted protocol as early as possible and convert to something you can authorize. A raw MLLP socket carries no identity. The moment those bytes land, wrap them in a message you can attribute, sign, and trace.

Two things go wrong here. First, people treat a network tunnel as authentication. A VPN tells you the packets came from the lab’s network. It does not tell you which system sent them, or that the sender was authorized to send that patient’s results. Second, people assume consumer VPN products fill the gap. Tools like NordVPN or Surfshark are fine for your own admin laptop on untrusted wifi, but they carry no business associate agreement and they are not a substitute for AWS Site-to-Site VPN or AWS Client VPN on a clinical path.

One piece of housekeeping that is cheap to do and expensive to skip: the business associate addendum. AWS makes this self-service through AWS Artifact, and it can be accepted across an entire organization. Only services on the current HIPAA Eligible Services Reference may hold PHI, and that list changes, so check it against the services you actually chose rather than your memory of it.

Boundary two: the landing zone

Raw messages land in S3. This part is well documented, so I will focus on the parts people configure once and never verify.

Make the bucket policy do the enforcing

Default bucket encryption means objects get encrypted. It does not mean a caller cannot write an object with different settings, and it does not stop a plaintext HTTP request. Two explicit denials cover the gap. The first protects the transport, the second protects the storage:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::lab-raw-inbound",
        "arn:aws:s3:::lab-raw-inbound/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::lab-raw-inbound/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id":
            "arn:aws:kms:REGION:ACCOUNT:key/KEY-ID"
        }
      }
    }
  ]
}

Use a customer managed KMS key, not the AWS managed one. The reason is not stronger cryptography, it is the key policy. A customer managed key gives you a second, independent authorization surface: even a principal with broad S3 permissions cannot read the object if the key policy does not grant them decrypt. That separation is what makes a blast radius argument credible to an assessor, and it is what makes crypto shredding possible later.

Your object keys are metadata, and metadata leaks

This is the one I would put on a poster. Object contents are encrypted. Object keys are not. They show up in CloudTrail data events, S3 server access logs, S3 Inventory reports, bucket listings, Lambda event payloads, and every console screenshot anyone pastes into a ticket.

# Leaks PHI into six places that are not the bucket
raw/2024/ORU_MRN4471902_SMITH_JOHN_A1C.hl7

# Same object, no identifiers in the key path
raw/dt=<partition>/src=lis01/msg=01HXYZ...ULID.hl7

Use an opaque, sortable identifier and keep the mapping from identifier to patient inside an encrypted store you actually control. Partition on ingest date and source system, never on anything derived from the patient. This one decision removes more PHI surface than any other single change in the pipeline.

Immutability, and the trap inside it

Clinical laboratories in the United States operating under CLIA have record retention obligations set out in 42 CFR 493.1105. The floor for test reports is at least two years after the date of reporting, with pathology test reports at ten years, and several state requirements sit above the federal floor. Retention is therefore not something you invent; go and read the section, then check your state.

S3 Object Lock gives you write-once-read-many enforcement, and it is the right tool. But understand the two modes before you commit, because one of them is genuinely irreversible.

• Governance mode blocks deletion for most principals, but anyone holding s3:BypassGovernanceRetention can override it by sending the x-amz-bypass-governance-retention:true header. Note the S3 console includes that header by default.
• Compliance mode cannot be overridden by anyone, including the account root. Per AWS documentation, the only way to delete an object under compliance retention before its date expires is to delete the AWS account.

A typo in a retention date under compliance mode is permanent, and you pay to store the mistake for its full term. Object Lock also has to be enabled at bucket creation, alongside versioning. Start in governance mode, prove the retention values against real traffic, then move to compliance mode deliberately rather than as a default.

# Indefinite hold for a specific object version, e.g. under litigation
aws s3api put-object-legal-hold 
  --bucket lab-raw-inbound 
  --key raw/dt=.../msg=01HXYZ.hl7 
  --legal-hold Status=ON

Boundary three: where PHI escapes into your observability

Here is the section that justifies the post. Everything above is on the compliance checklist. This is the part that is invisible until an audit or an incident makes it visible.

Your operational plane is a PHI store you did not declare. Specifically:

• Application logs. One print(record) left in a parser during debugging writes patient demographics into CloudWatch Logs, where the default is often no expiry at all.
• Exception traces. Unhandled parse errors frequently include the offending segment in the message. That segment is the PHI.
• Step Functions execution history. If you pass message bodies between states as payloads, that content is retained in the execution record and readable by anyone who can view executions.
• Dead letter queues. The whole point of a DLQ is to keep the failed message. A DLQ full of unparseable HL7 is a PHI archive with a different access policy than the bucket you designed so carefully.
• Third-party monitoring. Anything shipped to an external platform leaves your boundary. Grafana Cloud, Datadog and equivalents are perfectly reasonable choices for infrastructure telemetry, but a log line containing PHI going to a vendor without an executed BAA is a disclosure, not a metric.
• Notifications. An SNS topic that emails an on-call engineer “failed to process record for J. Smith, DOB …” has just sent PHI to a mail server nobody reviewed.

The pattern that fixes it: pass pointers, not payloads

Adopt a single rule and enforce it in review: PHI moves by reference between components, never by value.

1. The ingest function writes the raw message to the encrypted bucket and emits only an opaque object key plus a message ID.
2. Every downstream state, queue and event carries that pointer. Nothing carries the record body.
3. Error handling logs the pointer and a failure class, never the content that failed.
4. A DLQ therefore contains pointers to failures, and reprocessing means re-reading from the bucket under the same key policy as everything else.

The cost of this pattern is a slightly noisier debugging experience. You cannot read the failing record straight out of the queue. You have to go and fetch it, with credentials, and that fetch is logged. That inconvenience is the control working.

Then close the log surface itself

Log groups created implicitly by Lambda and other services often retain data indefinitely unless you set a policy. Find the ones nobody configured:

# List log groups with no retention configured
aws logs describe-log-groups 
  --query 'logGroups[?retentionInDays==null].logGroupName' 
  --output text

# Set an explicit retention on one of them
aws logs put-retention-policy 
  --log-group-name /aws/lambda/hl7-ingest 
  --retention-in-days 30

Run the first command as a scheduled check, not a one-off. New functions create new log groups, and the default comes back every time someone deploys.

Amazon Macie is worth pointing at your buckets as a detective backstop. It will not stop a leak, but it will tell you when something started landing where it should not.

Boundary four: from identified data to analytics

At some point someone wants to query the data. This is the crossing where teams are most likely to convince themselves they have done something they have not.

Amazon Comprehend Medical will detect PHI entities in unstructured clinical text through its DetectPHI operation, returning entity types such as name, address, identifier and date, each with a confidence score. It is a genuinely useful tool for narrative fields: pathology comments, specimen notes, microbiology free text.

aws comprehendmedical detect-phi 
  --text "Specimen received from the referring clinic on the stated date."

Now the part the marketing pages underplay, straight from the AWS developer guide: the entities Comprehend Medical detects do not map one to one to the identifier list specified by the Safe Harbor method, and AWS explicitly recommends additional human review or other methods to confirm accuracy for compliance use cases.

Read that as: PHI detection is not de-identification. Under the HIPAA Privacy Rule, de-identification has two defined routes, Safe Harbor and Expert Determination. An NLP model with confidence scores is an input to either, not a substitute for either. If your analytics tier is meant to hold de-identified data, someone has to own that determination, and it will not be your pipeline.

What works structurally:

• Put the analytics tier in a separate AWS account with its own KMS key. Not a separate bucket. A separate account, so the blast radius argument is enforced by IAM boundaries rather than by convention.
• Make the flow one directional. The de-identification job reads from the identified side and writes to the analytics side. Nothing in the analytics account holds permissions pointing back.
• Replace identifiers with pseudonyms rather than deleting them, keeping the crosswalk in the identified account. This preserves your ability to join across results and to re-identify under a documented process, which the research use case usually needs.
• Watch quasi-identifiers. Rare test panels, unusual reference ranges and precise timestamps re-identify people in small populations even after the obvious fields are gone. This is a statistics problem, not an IAM problem.
• Use Lake Formation or equivalent for column and row filtering if analysts need partial access, so the restriction lives with the catalog rather than in each query tool.

Boundary five: sending results back out

Delivery is where clinical correctness and security pull against each other, and security usually loses quietly.

The correction problem is the sharp edge. HL7 result messages carry a status that distinguishes preliminary, final and corrected results. If your delivery layer is idempotent on message ID alone, a corrected result with a new ID for the same observation will happily land beside the original instead of superseding it. Two conflicting values, both marked delivered, nobody alerted.

Build the idempotency key from the identity of the observation, which typically means the placer or filler order identifier combined with the specific observation identifier, and carry the result status as a first-class field so a correction overwrites rather than appends. Keep every version, expose the current one.

For the outbound path itself:

• Presigned URLs are convenient for report PDFs and they are also a bearer credential. Anyone holding the link has the document. Keep expiry short and treat generation as an auditable event.
• If you expose a provider-facing portal, put a WAF in front of it. AWS WAF and Cloudflare are both credible here; the deciding factor is usually where the rest of your edge already lives, not the rule engines.
• Never email a result. Email a notification that a result is available, behind authentication. The distinction sounds pedantic right up until someone forwards a mailbox.

Boundary six: deletion, and the conflict nobody plans for

Two obligations point in opposite directions. Retention rules say keep the record. Privacy rights and internal policy say be able to remove data. If you set compliance-mode Object Lock across everything, you have chosen one side without noticing.

Resolve it by classifying before you lock. Records that fall under laboratory retention obligations go into the immutable tier with a retention period derived from the applicable rule. Derived artifacts, caches, intermediate parquet, analytics extracts and enrichment outputs do not belong there. They belong in buckets with lifecycle rules, and they are what you actually delete.

Crypto shredding covers the middle ground: if a data set is encrypted under a key used for nothing else, scheduling that key for deletion renders the ciphertext unrecoverable without touching the objects. It is clean, it is verifiable, and it only works if you planned the key granularity up front. Retrofitting per-tenant or per-cohort keys onto a pipeline that used one key for everything is a full re-encryption exercise.

Do not forget the endpoints. Instrument workstations, the interface engine box, and the laptop somebody used to test the parser all accumulate copies. Cloud controls do nothing for physical media, and secure erase tooling such as O&O SafeErase exists for exactly that step in a decommissioning runbook.

Troubleshooting a medical lab data pipeline on AWS

Messages arrive but nothing appears downstream

Check the DLQ first, then check whether the DLQ itself has a consumer. A surprising number of pipelines have a correctly configured dead letter queue that nobody monitors, so failures accumulate silently and the only symptom is a gap in the data. Alarm on DLQ depth greater than zero, not on a threshold.

Access denied from a Lambda that has the right IAM policy

With a customer managed KMS key, IAM permission on S3 is only half the grant. The function’s role also needs decrypt permission on the key, and the key policy has to allow it. If you are crossing accounts, both the key policy and the role policy must permit the action. This is the intended behavior of the two-lock design, and it is the single most common support question on any pipeline built this way.

Deletion fails on an object you are certain you should be able to delete

Check for a legal hold before you check retention. A legal hold has no expiry and is independent of the retention period, so an object can be past its retention date and still undeletable. Also confirm which mode the retention uses, because governance and compliance produce the same error to a caller without bypass permission.

Counts do not reconcile with the LIS

Compare on observations, not messages. One message can carry several results, corrections create additional messages for the same observation, and a naive message count will diverge from the LIS by exactly the amount that matters clinically. Reconcile daily and alert on drift rather than investigating at quarter end.

Timestamps drift between systems

HL7 v2 timestamps do not always carry a timezone offset, and analyzers are often set to local time with no daylight saving handling. Capture the offset at ingest from the source system configuration and store everything in UTC with the original string preserved. Recovering an ambiguous timestamp after the fact is unpleasant, and around a daylight saving transition it may not be possible at all.

Common mistakes

• Treating “HIPAA eligible” as “compliant.” Eligibility means AWS will cover the service under the BAA. Configuration is still entirely yours.
• Putting identifiers in object keys, queue names, or log group names. Encrypted contents, plaintext metadata.
• Enabling compliance-mode Object Lock before validating retention values. There is no undo, and no support ticket that fixes it.
• Assuming automated PHI detection equals de-identification. AWS itself says the entity list does not map one to one to Safe Harbor identifiers.
• Running the analytics tier in the same account as identified data. A bucket boundary is a policy away from collapsing. An account boundary is not.
• Shipping application logs to an external platform without checking what the log lines contain. Test it by grepping your own log group for a known test patient name.
• Ignoring corrected results until an analyst notices. Amendment handling is a day-one requirement, not a phase two feature.

Best practices worth the effort

• Write the data flow diagram first and mark every trust boundary crossing. Every arrow that crosses one needs a named control. This exercise finds more problems than any scanner.
• Encode the eligible service list as policy. Service control policies at the organization level stop someone from putting PHI into a service you never assessed.
• Keep VPC endpoints on the PHI path. Interface and gateway endpoints keep traffic to AWS services off the public internet and give you an endpoint policy as an extra choke point. Check which of your chosen services offer one.
• Build with synthetic HL7 from day one. Nobody should need production PHI to develop a parser. A cheap VPS from Contabo or InterServer is fine for a synthetic-data development box, and keeping that environment entirely outside the PHI boundary is the point.
• Alarm on absence. A feed that stops is more dangerous than a feed that errors, because errors are loud. Alert when expected message volume for a source drops below its floor.
• Test restores, not backups. Restore into an isolated account, confirm the KMS grants work there, and document how long it took.
• Review CloudTrail for data events on the PHI buckets. Object-level logging costs money and is the only record of who read what.

FAQ

Do I need a BAA with AWS before I start building?

Before PHI touches the environment, yes. You can architect and test with synthetic data first. AWS provides the business associate addendum through AWS Artifact as a self-service acceptance that can be applied across an organization, and only services listed on the HIPAA Eligible Services Reference may process, store or transmit PHI.

Should I use AWS HealthLake or build on S3, Glue and Athena?

HealthLake is a managed FHIR datastore with FHIR APIs, medical NLP and query built in, and it is the shorter path if your consumers speak FHIR. It does not natively ingest HL7 v2; AWS points to partner tooling or your own transformation for non-FHIR input. If your consumers are analysts with SQL and the data is structured lab results rather than clinical documents, an S3 and Athena lakehouse is simpler and easier to keep inside a boundary you control. Decide on the consumer, not the source.

How long do I have to keep laboratory records?

For CLIA-regulated laboratories in the United States, 42 CFR 493.1105 sets the federal floor, including at least two years for test reports after the date of reporting and at least ten years for pathology test reports. State law frequently exceeds these minimums, so the applicable period is whichever is longer for your jurisdiction. Read the regulation and confirm with your compliance lead before you encode a number into a retention policy.

Can Amazon Comprehend Medical de-identify data for me?

It can detect PHI entities and give you confidence scores, which is a strong starting point for redaction workflows. It does not by itself satisfy HIPAA’s de-identification standard. AWS documents that the detected entities do not map one to one to the Safe Harbor identifier list and recommends human review for compliance use cases. Treat it as a detector feeding a documented Safe Harbor or Expert Determination process.

Is S3 Object Lock compliance mode required for lab records?

Not inherently. The argument for compliance mode is a threat model in which a compromised or malicious administrator could remove retention, not a regulation that names the mode. Governance mode plus tightly controlled bypass permissions and strong audit logging is a defensible position for many laboratories. Choose deliberately and write down the reasoning, because compliance mode commits you to the storage cost for the full term with no exit.

Where does PHI most often leak in a pipeline that looks correctly configured?

Logs, dead letter queues, workflow execution histories, notification messages and object key names. All five are metadata surfaces that sit outside the storage layer everyone reviews. Grep your own log groups for a known test patient identifier and you will find out quickly whether your pipeline has the problem.

Do I need a dedicated AWS account just for the lab pipeline?

Separate accounts for identified and de-identified data are worth it even at small scale, because the boundary is then enforced by IAM rather than by naming conventions. Whether the PHI workload also needs isolating from your other production workloads depends on who holds admin access there. If the answer is “broadly the same people,” separate it.

The one thing to take away

A secure medical lab data pipeline on AWS is not defined by the encryption on its storage layer. Encryption is the easy part, and it is the part everyone gets right. What separates a design that survives an audit from one that does not is whether PHI can travel by value through your operational plane: into logs, into queue bodies, into workflow histories, into filenames, into alert emails.

Move PHI by reference, keep identifiers out of every metadata surface, and put an account boundary between identified and de-identified data. Do those three things and the rest of the compliance work becomes documentation rather than redesign.

Work with me on healthcare data pipelines

I design and review AWS data platforms for regulated workloads, and lab data is a specific enough problem that generic cloud advice tends to miss it. Things I can help with:

• Mapping every trust boundary in an existing pipeline and finding the PHI surfaces nobody documented, including logs, queues and object key schemes.
• Designing HL7 v2 or file-drop ingestion into S3 with KMS key separation, VPC endpoints and bucket policies that enforce rather than suggest.
• Building the identified and de-identified account split, including the one-way transformation job and the pseudonym crosswalk.
• Getting amendment and correction handling right so corrected results supersede rather than duplicate, all the way to the analytics tables.
• Retention and deletion design: Object Lock modes, lifecycle policies, crypto shredding key granularity, and the runbook that ties them together.
• Observability that is useful without being a disclosure risk, including CloudWatch retention hygiene and safe alerting patterns.

If you have an architecture diagram, a Terraform plan, or a redacted sample of the messages you are receiving, send it over and I will tell you what I would change first.

The post Where PHI Escapes: Building a Secure Medical Lab Data Pipeline on AWS appeared first on John Nessime.

That is the failure mode that defines this problem. An insurance claims processing pipeline on AWS almost never falls over in the way you designed it to fall over. It succeeds. It emits well-formed JSON. It hands a number to a payment system, and the number is wrong, and nothing in the pipeline had any reason to think otherwise.

This post is organized by failure family rather than by service. I’ll walk through the six ways these pipelines go quietly wrong, what each one costs, and what the fix actually looks like in Textract, Bedrock, Step Functions and S3. There’s a troubleshooting section, the mistakes I see repeated, and an FAQ at the end.

The shape most claims pipelines end up with

Before the failure families make sense, the skeleton. Almost every serverless claims pipeline lands on roughly the same set of stages, whatever the vendor deck calls them:

1. Intake. A document lands in S3 from a portal upload, an SFTP drop, or a mail scanning vendor. An S3 event or EventBridge rule starts an execution.
2. Classification. Work out what the packet actually contains. A first notice of loss, a CMS-style claim form, a police report, an itemized bill, forty pages of photographs.
3. Extraction. Pull the fields you need. Amazon Textract for OCR, forms and tables, or Amazon Bedrock Data Automation with a blueprint that names the fields directly.
4. Validation. Check the extracted values against business rules, policy data, and each other.
5. Routing. Straight-through processing, human review, or rejection with a reason.
6. Persistence and audit. The claim record, the extraction artifacts, and enough evidence to explain a decision months later.

Nothing controversial there. AWS publishes an open-source GenAI IDP Accelerator that implements exactly this shape, with a Bedrock Data Automation mode and a Textract-plus-foundation-model pipeline mode, and it’s a reasonable place to start reading. The interesting part is not the boxes. It’s what happens between them.

Failure family one: the field that was never there

This is the one that pays out the wrong number, and it is worth more attention than everything else in this post combined.

Every extraction service gives you confidence scores. So the obvious design is a gate: if every field scores above some threshold, approve automatically; if anything falls below, send it to a human. That gate is sound reasoning applied to the wrong population.

A confidence score only exists for a value that came back. When the extractor doesn’t find a field at all, there is no low score to catch, because there’s nothing to score. The gate iterates over four returned fields, finds all four above threshold, and reports a clean pass. The fifth field, the one that determines coordination of benefits or the deductible offset, is simply absent from the response. Downstream code treats absent as zero, or as null, or as “not applicable,” and the claim goes through.

The fix is structural, not statistical. Validate presence against a schema before you validate confidence, and treat the two as separate gates with separate outcomes:

# Two gates, not one. Presence first, then confidence.
# 'extracted' is the flattened field map from Textract Queries
# or a Bedrock Data Automation blueprint result.

REQUIRED = {
    "claim_number",
    "date_of_service",
    "billed_amount",
    "member_id",
    "secondary_payer_indicator",
}

def gate(extracted, scores, threshold=0.95):
    missing = REQUIRED - set(extracted)
    if missing:
        # Never silently default. This is a routing decision.
        return "HUMAN_REVIEW", {"reason": "missing_fields",
                                "fields": sorted(missing)}

    weak = [f for f in REQUIRED if scores.get(f, 0.0) < threshold]
    if weak:
        return "HUMAN_REVIEW", {"reason": "low_confidence",
                                "fields": sorted(weak)}

    return "STRAIGHT_THROUGH", {}

Two details matter here. The set difference is computed against a declared schema, not against whatever keys happen to be in the response, so an absent field becomes a first-class routing reason. And scores.get(f, 0.0) defaults to zero rather than to a passing value, so a field that arrives without a score fails closed.

If you’re on Textract Queries, there’s a second reason to be explicit: Queries let you attach an alias to each question, which means your schema keys are yours rather than whatever label happened to be printed on the form. That’s the difference between “the field is missing” and “the field moved and we didn’t notice.”

Failure family two: confidence scores that answer a different question

Assume you’ve fixed presence. The next trap is what the confidence number is measuring.

OCR confidence is a statement about characters. It says the model is highly sure those pixels read 1,240.00. It is not a statement that 1,240.00 is the billed amount rather than the allowed amount from the box directly above it, or the prior balance from a remittance summary that happened to be stapled into the same packet. Read it as a legibility score, because that’s closer to what it is.

Bedrock Data Automation narrows this gap: blueprints define fields semantically, confidence scores come with bounding boxes, and the visual grounding lets you point at the region a value came from. Textract Queries narrow it too, by asking a question rather than harvesting a label. Neither eliminates the problem, because a high-confidence read of the wrong region still scores high.

What actually catches this is cross-field invariants. They cost almost nothing and they fail for reasons a human can read:

• Arithmetic. Line items sum to the claimed total. If they don’t, one of the two is wrong and you don’t yet know which.
• Temporal. Date of service falls inside the policy period and before the date of submission. A service date after the submission date is a parsing error nine times out of ten.
• Referential. The member or policy identifier resolves against your system of record. An identifier that matches the format but not a real record is a strong signal you read the wrong box.
• Range. Amounts within a plausible band for the claim type. Not a fraud model, just a tripwire for a decimal point that moved.
• Page provenance. Fields that must come from the same page or the same document within the packet. Bounding box data makes this checkable rather than assumed.

An invariant failure is more useful than a low score, because it names a contradiction. “Line items sum to 1,180 but the claimed total reads 1,240” is something a reviewer resolves in seconds. “Confidence 0.91” is something a reviewer stares at.

Failure family three: the claim that stops halfway

Claims documents are multi-page packets, so you’ll be using Textract’s asynchronous operations. That means jobs, notifications, and a whole class of orchestration bugs that only show up under load or after a weekend.

The asynchronous pattern is: call StartDocumentAnalysis, get a JobId back, and let Textract publish completion to an SNS topic you nominate. A request looks like this:

{
  "DocumentLocation": {
    "S3Object": { "Bucket": "claims-intake", "Name": "packets/abc123.pdf" }
  },
  "FeatureTypes": ["FORMS", "TABLES"],
  "ClientRequestToken": "abc123-v1",
  "JobTag": "fnol-packet",
  "NotificationChannel": {
    "SNSTopicArn": "arn:aws:sns:REGION:ACCOUNT:textract-complete",
    "RoleArn": "arn:aws:iam::ACCOUNT:role/TextractPublishRole"
  },
  "OutputConfig": {
    "S3Bucket": "claims-extraction",
    "S3Prefix": "raw/"
  },
  "KMSKeyId": "alias/claims-cmk"
}

Three of those parameters are doing load-bearing work that is easy to skip.

ClientRequestToken is the idempotency token. Reuse the same token and you get the same JobId back instead of a second job. Derive it from the document, not from the invocation, and a Lambda retry or a duplicated S3 event stops turning into a duplicate charge and a duplicate claim record.

OutputConfig writes results into a bucket you control. Without it, results stay internal to Textract and the only way to read them is the Get operations, which have their own throttling limits. Under concurrency those limits become the bottleneck: you end up polling more jobs than you’re allowed to poll, backing off, and watching end-to-end latency climb for reasons that have nothing to do with the documents. Writing to S3 sidesteps the whole path.

JobTag shows up in the completion notification. In a mixed pipeline where the same topic carries first notice of loss packets, itemized bills and ID documents, that tag is what lets the notification handler route without a lookup.

One expiry to plan around: a Textract JobId is only valid for seven days. If your retry story is “requeue it and someone will look on Monday,” a bad weekend turns recoverable failures into full reprocessing. Persist the S3 output location, not the job identifier.

Callbacks that never come back

Human review means pausing a workflow for hours or days, which in Step Functions means the callback pattern. You append .waitForTaskToken to the resource ARN, pass $$.Task.Token into the payload, and the execution parks until something calls SendTaskSuccess or SendTaskFailure with that token.

The trap is that a callback task with no timeout waits until the execution itself hits its quota, and Standard workflow executions can run for up to a year. A reviewer who leaves, a review UI that drops the token, a queue consumer that crashes after reading the message and before writing it to the review table: all of these produce an execution that is neither failed nor finished. It just sits there. Nobody alerts on it because nothing broke.

"AwaitAdjusterDecision": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "enqueue-review-task",
    "Payload": {
      "claimId.$": "$.claimId",
      "taskToken.$": "$$.Task.Token"
    }
  },
  "TimeoutSeconds": 259200,
  "HeartbeatSeconds": 3600,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "EscalateStaleReview"
  }],
  "Next": "ApplyDecision"
}

TimeoutSeconds is the maximum total lifetime of the task regardless of heartbeats. HeartbeatSeconds is the maximum gap between SendTaskHeartbeat calls, so a review app that periodically confirms the item is still in someone’s queue will fail fast when that app dies, rather than at the outer limit. AWS’s own guidance is to set the heartbeat below the task timeout for exactly this reason: a heartbeat failure tells you the worker died, a timeout tells you the work took too long, and those are different incidents. Catch States.Timeout and route to a real state. An unhandled timeout is just a differently-shaped silence.

One constraint worth knowing before you design around it: the callback pattern requires Standard workflows. Express workflows support request-response integrations only, so no .waitForTaskToken and no .sync. If you split your pipeline into a fast Express path and a Standard review path, the boundary between them is where the token has to live.

Failure family four: the human review service you can no longer sign up for

This one catches people copying a reference architecture, and it’s the reason to read publication dates on IDP blog posts.

Amazon Augmented AI, known as A2I, was the managed answer to human-in-the-loop review. It plugged directly into Textract’s AnalyzeDocument, watched confidence conditions, and spun up review tasks with a worker UI for you. It appears in a great many architecture diagrams for claims and lending workflows.

Per the AWS documentation, SageMaker A2I is no longer open to new customers. Existing customers can keep using it, and AWS continues security and availability work, but no new features are planned. If you’re standing up a new account today, that diagram does not deploy.

Be fair about what that costs you, because A2I genuinely removed real work: the task assignment logic, the worker UI, the private workforce plumbing through Cognito, result consolidation. Rebuilding it means owning all of that. What you get back is that the review queue becomes yours, which in practice means you can put claim-specific context on the screen instead of a generic key-value editor. For adjusters that difference is not cosmetic.

A minimal replacement is not exotic:

• A DynamoDB table of review items, each holding the claim identifier, the extracted values, the bounding boxes, and the Step Functions task token.
• A small web app for reviewers that renders the page image with the boxes overlaid, so a reviewer confirms placement rather than retyping values.
• An API that writes the corrected values and calls SendTaskSuccess with the stored token.
• Authentication in front of it. Amazon Cognito if you want to stay inside AWS, or an identity-aware proxy such as Cloudflare Access if your reviewers are external adjusters you’d rather not create AWS identities for.
• A sweeper that finds review items older than your heartbeat window and escalates them.

If the review app is a small internal tool with no data residency requirement of its own, it doesn’t have to live in the same account or even the same provider. A modest VPS from a host like Contabo or InterServer running behind a Cloudflare Tunnel is a legitimate answer for a reviewer console that talks to AWS over scoped API credentials. Just be honest about what crosses that boundary, which brings us to the next family.

Failure family five: claim data in places nobody decided to put it

Claims documents carry protected health information, financial identifiers, and often photographs of people and property. The pipeline you drew has three or four places that data lives. The pipeline you deployed has a dozen.

The ones that get missed:

• Lambda logs. One print of an event payload during a debugging session, and CloudWatch Logs is now a claims repository with a different retention policy and a different access model.
• Dead letter queues. A DLQ holds the full failed message. If that message carries extracted values, your DLQ is regulated data, and it is usually the least governed thing in the account.
• Step Functions execution history. State input and output are visible in the console and the history API. Passing extracted fields between states puts them there.
• Intermediate extraction output. The bucket you pointed OutputConfig at holds raw OCR of the whole packet, often with a lifecycle policy nobody wrote.
• Model invocation logging. Bedrock can log inputs and outputs to S3 or CloudWatch. Useful for debugging, and another copy of everything.

The pattern that keeps this manageable is passing pointers, not payloads. States carry an S3 key and a claim identifier; the values themselves stay in one encrypted bucket with one lifecycle policy and one access policy. It makes debugging marginally more annoying and it makes the data map fit on a page.

On regulated workloads, check the current AWS HIPAA-eligible services list and your executed BAA for every service in the path, in the specific region you’re deploying to. Eligibility is per service and it changes. Textract has long been used for claims workflows on that basis, and Bedrock is listed as HIPAA eligible, but “I read a blog post” is not a control. Pull the list yourself before PHI touches anything.

Two smaller things worth deciding early. Reviewers working from home should reach the console over something better than the open internet, whether that’s a corporate tunnel, a business VPN account from a provider like NordVPN or Surfshark, or an identity-aware proxy. And if reviewers ever download claim documents locally, agree what happens to those files afterward, because a deleted file is not an erased file. Tools such as O&O SafeErase exist for exactly that gap on Windows endpoints.

Failure family six: paying twice for the same page

Document AI services bill per page. That single fact reshapes how you think about retries, because in most pipelines a retry is free and here it isn’t.

The expensive patterns are all shaped the same way. A poison document fails a downstream parser, gets requeued, and is re-extracted on every attempt. A batch job re-runs over an entire prefix instead of a delta. A misconfigured S3 event delivers twice. An operator reprocesses a day’s intake to fix a mapping bug in the transform stage, when the extraction stage was fine all along.

The structural fix is separating extraction from interpretation. Extract once, write the raw result to S3 keyed by a content hash of the document, and let every downstream stage read from that. When the mapping bug shows up, you re-run interpretation over stored output and pay nothing. Combined with ClientRequestToken, most accidental double-charges disappear.

Also route the packet before you extract it. Forty pages of accident photographs do not need forms and tables analysis. Classification is cheaper than extraction, and page-level routing is often the single largest lever on the bill.

For attributing that spend, cost allocation tags on the buckets and functions give you the AWS-native view, and platforms like Vantage or CloudZero are worth a look if you need per-claim or per-client unit costs rather than per-service totals. Whatever you use, the metric that matters is cost per claim processed, split by straight-through versus reviewed. Those two numbers tell you whether the automation is earning its keep.

Troubleshooting an insurance claims processing pipeline on AWS

Symptoms you’ll actually see, and where to look first.

• Executions succeed but downstream amounts are wrong. Check whether required fields are present, not just confident. Diff the schema against the response keys for a sample of recent claims. This is failure family one until proven otherwise.
• Executions stuck in Running for days. A callback task with no timeout. List running executions ordered by start time and look for the state name of your review task.
• Throttling on the extraction stage under load. If you’re polling Get operations, move to OutputConfig and SNS notification and stop polling. If you’re already there, check the start-operation limits rather than assuming the whole service is slow.
• The same claim appearing twice. Look for a missing or per-invocation ClientRequestToken, and check whether your S3 event handler is idempotent. Delivery is at-least-once.
• Extraction quality dropped for one document type. Usually the form changed, not the model. Compare bounding boxes for the affected field against an older sample. If the box moved, that’s a layout change, and query aliases or a blueprint update is the fix.
• Review queue growing faster than reviewers clear it. Break the routing reasons apart. If most items are low confidence on one field, that’s an extraction problem wearing a staffing problem’s clothes.
• SNS notification arrives, handler can’t find the results. Confirm the notification role has permission to publish and the handler is reading the S3 prefix rather than calling Get with an expired job identifier.

Common mistakes

• Gating only on confidence, so a missing field is indistinguishable from a clean extraction.
• Defaulting absent values to zero or null in the transform layer instead of raising a routing decision.
• Copying an architecture diagram that includes A2I into a new AWS account.
• Callback tasks with no TimeoutSeconds and no heartbeat.
• Passing extracted claim values through Step Functions state rather than passing an S3 pointer.
• Treating a single global confidence threshold as adequate for every field. A name and a dollar amount do not carry the same downstream risk.
• Running forms and tables analysis over every page of a packet including the photographs.
• No metric for straight-through rate, so nobody notices when it quietly drops.

Best practices worth the effort

• Declare the schema, then validate presence, then confidence, then invariants. Four gates, four distinct rejection reasons, four things a reviewer can act on.
• Set per-field thresholds by consequence. Get the payable amount wrong and money moves. Get a street suffix wrong and a letter is slightly odd.
• Keep a labeled regression set. A few dozen real packets with known-correct values, run on every blueprint or query change. Bedrock Data Automation can use ground-truth examples to refine blueprint instructions, which only works if you maintain the ground truth.
• Store bounding boxes alongside values. They make review faster and they turn “quality dropped” from a guess into a comparison.
• Alarm on rates, not just errors. Straight-through rate, review-queue age, and cost per claim. A pipeline that stops approving anything is broken even though nothing threw.
• Make every stage idempotent on a content hash. Reprocessing is normal. It should be safe and cheap.
• Instrument the pipeline like a pipeline. CloudWatch covers the AWS surface; if you’re consolidating with on-premises claims systems, a platform like Grafana Cloud gives you one place to correlate both sides.

Frequently asked questions

Should I use Amazon Textract or Bedrock Data Automation for claims extraction?

Textract is the sharper tool when your documents are standardized forms and you want deterministic OCR with forms, tables and targeted queries. Bedrock Data Automation is stronger on mixed packets, because it splits along logical document boundaries, classifies each part, and applies a blueprint per document type, with confidence scores and visual grounding on the output. Claims intake is usually mixed packets, which tilts toward Data Automation, but the honest answer is to run both against a sample of your real documents. The evaluation costs a day and it decides your architecture.

What replaces Amazon A2I for human review?

For new AWS accounts, a custom review path: a queue or table of review items, a reviewer UI, and the Step Functions callback pattern to resume the workflow. It’s more code than A2I but not a large amount, and it gives you a review screen designed around claims rather than around generic key-value pairs. Existing A2I customers can continue as they are, though building on a service with no planned features is a decision to make deliberately rather than by default.

What straight-through processing rate should I expect?

Anyone quoting you a number without seeing your documents is guessing. It depends almost entirely on document quality and how many fields you require. What’s reliable is the method: measure your current rate, split failures by reason, and fix the largest reason. Requiring one rarely-present field can dominate everything else, and that’s a policy decision as much as an engineering one.

How do I keep an insurance claims processing pipeline on AWS HIPAA-aligned?

Start from the AWS HIPAA-eligible services list and an executed BAA, and confirm eligibility for each service in your specific region. Then do the unglamorous work: customer-managed KMS keys, no PHI in logs or state payloads, scoped IAM roles per stage, VPC endpoints where the service supports them, retention policies on every bucket and queue including dead letter queues, and CloudTrail configured so you can answer who accessed which claim. Eligibility is permission to build; the controls are yours.

Can I run the whole pipeline with Step Functions Express workflows?

Not the part that waits for a human. Express workflows support request-response integrations only, so the callback pattern requires Standard. A common split is Express for the high-volume deterministic stages and Standard for anything holding a task token, with the two connected by an event or a queue.

How do I stop duplicate claims from duplicate events?

Treat every trigger as at-least-once. Derive an idempotency key from the document itself, usually a content hash plus a version marker, pass it as ClientRequestToken to the extraction call, and use it as the conditional write key when you create the claim record. Then a duplicate event is a no-op rather than a second claim.

Is it worth starting from the AWS GenAI IDP Accelerator?

As a reference for structure and as a way to get a working pipeline in front of stakeholders quickly, yes. As a production system you inherit wholesale, be careful: you’re adopting someone else’s opinions about classification, review and storage, and you’ll be reading that code anyway the first time something behaves oddly. Read it, borrow the patterns, own what you deploy.

The one thing to take away

An insurance claims processing pipeline on AWS is not hard to build. Textract, Bedrock Data Automation, Step Functions and S3 will get you a working pipeline in a couple of weeks. What’s hard is making it fail in ways you can see.

Every expensive failure in this space shares one shape: the pipeline had no opinion about what it did not receive. A field that didn’t come back scored nothing, a callback that never fired errored nothing, a duplicate event failed nothing. If you take one design rule from this, take that one. Declare what a complete claim looks like, check for its absence explicitly, and route anything incomplete to a human with a reason attached.

Green executions are not evidence. Reconciled numbers are.

Need help with your claims pipeline?

I work with teams building document-heavy workflows on AWS, and claims pipelines are one of the places where a small amount of design care prevents a large amount of reconciliation work. Things I can help with:

• Reviewing an existing extraction pipeline for silent-failure paths, particularly missing-field handling and default values in the transform layer
• Designing the routing logic: schema gates, per-field thresholds, cross-field invariants, and the escalation rules that sit behind them
• Building a human review path with the Step Functions callback pattern, including timeouts, heartbeats and a sweeper for stale tasks
• Running a structured evaluation of Amazon Textract against Bedrock Data Automation on your actual documents, with a labeled regression set you keep afterward
• Tracing where claim data actually lands across logs, queues, execution history and intermediate buckets, then shrinking that footprint
• Instrumenting straight-through rate, review-queue age and cost per claim so regressions surface before finance finds them

If you’d like a second opinion, send me a state machine definition, a sample extraction response with the values redacted, or the routing code that decides what goes to review. That’s usually enough to spot the gap.

The post Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly appeared first on John Nessime.

Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify’s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.

That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.

That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn’t there.

This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.

What the pipe actually looks like

Four moving parts, and only two of them live in your account.

1. A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.
2. Shopify creates a partner event source inside your AWS account, in the region you nominated.
3. You associate that source with a partner event bus. This is the step everyone forgets.
4. Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.

The ARN trips people up more than anything else in the setup. Shopify wants the event source ARN, not the event bus ARN. They look similar and only one of them works:

# Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/<id>/<source-name>

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/<id>/<source-name>

Associating the source is a single call, and both the name and the source name are the same string:

# Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --event-source-name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge’s semantics, not the store’s.

Failure family one: events that never existed

This is the one from the opening, and it is the most expensive because it is completely silent on both sides.

Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.

The same class of hole opens up in two other ways:

• Region mismatch. The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.
• Environment drift. A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.

The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls describe-event-source and alarms if State is anything other than ACTIVE costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.

The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on MatchedEvents hitting zero, not just on errors.

Failure family two: events that arrive twice, or backwards

EventBridge is at-least-once. Shopify’s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.

The duplicate is the obvious one. The same orders/create arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts X-Shopify-Webhook-Id into detail.metadata, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.

The out-of-order case is the one that costs you money quietly. An orders/updated carrying a cancelled status arrives before the orders/updated carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.

The envelope carries what you need for this too. detail.metadata includes X-Shopify-Triggered-At, and the resource in detail.payload carries its own updated_at. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.

The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.

One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.

Failure family three: rules that match nothing

Every Shopify event, regardless of topic, arrives with the same detail-type. That single fact invalidates the routing instinct most people bring from AWS service events.

{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/<id>/<source-name>",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}

Two things to take from that envelope. The resource body is nested under detail.payload, not at the top of detail, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in detail.metadata, which is where all your routing has to happen.

// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}

Do not deploy a pattern you have not tested against a real envelope. test-event-pattern answers in a second and saves an afternoon:

aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json

One rule or thirty?

There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.

What you give up is per-topic visibility. MatchedEvents, FailedInvocations and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.

The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.

Failure family four: events that arrive and die at the target

By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.

First, a whole class of errors gets no retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.

Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.

aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/<id>/<source-name>" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'

Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.

The DLQ permission trap

This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through PutTargets — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.

{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}

The metric that catches this is InvocationsFailedToBeSentToDlq. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.

Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.

Failure family five: the bill

EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.

This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.

Three levers, roughly in order of how much they return:

• Trim at the subscription. Shopify’s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.
• Subscribe to fewer topics. Broad topics like orders/updated fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.
• Archive selectively, and set retention. Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.

One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.

The layer nobody builds until they need it

Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that reached the bus. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.

For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.

• Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.
• Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.
• Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.
• Emit the drift as a metric, not just a log line. “Orders in Shopify but not in our store, last hour” is a graph worth putting on a dashboard, and it should normally read zero.

The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.

Troubleshooting by symptom

Work these in order. Each one is cheap and rules out a whole branch.

Nothing is arriving at all

1. Run describe-event-source. If State is not ACTIVE, stop here. Everything published so far is gone and you need the reconciliation path.
2. Confirm the region of the bus matches the region in the source ARN.
3. List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.
4. Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.
5. Check MatchedEvents on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.

Some topics arrive, order or customer topics do not

This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.

The rule matches but the target does nothing

Compare MatchedEvents against SuccessfulInvocationAttempts on the rule. A gap sends you to FailedInvocations and to the DLQ. Check the target’s resource policy, and check InvocationsFailedToBeSentToDlq before you trust that the DLQ is catching anything.

Events arrive, but late

Look at ThrottledRules and at IngestionToInvocationSuccessLatency. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.

Common mistakes

• Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.
• Registering the event bus ARN instead of the event source ARN, then debugging Shopify’s rejection for an hour.
• Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under detail.payload.
• Routing on detail-type. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.
• Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.
• Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.
• Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.
• Skipping reconciliation because the pipeline “works”. It works right up until it doesn’t, and that is precisely when you need the other path.

Best practices for streaming Shopify events into AWS

• Alarm on the event source state and on MatchedEvents reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.
• Dedupe on X-Shopify-Webhook-Id and version-check on X-Shopify-Triggered-At. Two mechanisms, two problems.
• Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.
• Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.
• Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.
• Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.
• Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.
• Build reconciliation before your first peak trading period, not after your first missing-order ticket.

Frequently asked questions

Do I still need to verify the HMAC signature on the EventBridge path?

No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.

Should I use EventBridge or plain HTTPS webhooks?

HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.

Can I use one partner event source for multiple stores?

Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by X-Shopify-Shop-Domain in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.

Why do I get duplicate order events even though nothing failed?

Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several orders/updated events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.

What happens to events published while my consumer is broken?

They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.

Can I archive and replay Shopify events?

Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.

Does this work the same way for BigCommerce or commercetools?

The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.

The one thing worth remembering

Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.

The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.

Need a hand with your event pipeline?

Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:

• Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.
• Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.
• Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.
• Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.
• Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.
• Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.

If you have a rule pattern that isn’t matching, a DLQ that’s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I’ll tell you what I’d look at first.

The post Streaming Shopify Events into AWS Without Losing Orders appeared first on John Nessime.