The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.

That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.

Why Amazon Textract data extraction is not just OCR

Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of Block objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.

That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the KEY_VALUE_SET block with entity type KEY, follow its VALUE relationship to another block, then follow that block’s CHILD relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.

Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.

Synchronous or asynchronous: the choice is made for you

People reach for AnalyzeDocument first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at one page. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.

One page. That single constraint decides your architecture. Any real contract goes through StartDocumentAnalysis and GetDocumentAnalysis, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.

Three other constraints from the same quota page are worth knowing before you promise anything to a client:

• Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.
• Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.
• Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.

Pass a ClientRequestToken on submission. It is an idempotency token: the same token returns the same JobId instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.

import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)

The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda’s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.

The result set you never finished reading

This is the one that cost the termination clauses. GetDocumentAnalysis returns results in pages. When there are more blocks than fit in one response, the response carries a NextToken and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. JobStatus still reports success, because the job did succeed. Your code just stopped reading.

Short test documents fit in a single response. That is exactly why this survives testing and dies in production.

def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks

Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.

Choosing feature types without paying for all of them

AnalyzeDocument and StartDocumentAnalysis take a FeatureTypes list: FORMS, TABLES, QUERIES, SIGNATURES and LAYOUT. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.

• FORMS finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.
• TABLES reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.
• QUERIES lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.
• SIGNATURES returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.
• LAYOUT groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.

A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.

Reading the block graph without losing fields

Here is the failure that produces silent empty strings. A key-value pair’s value is not always text. If the value is a tick box, the value block’s children include a SELECTION_ELEMENT block, and that block has no text at all. It has a SelectionStatus of SELECTED or NOT_SELECTED. Concatenate only WORD children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.

def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

Build block_map once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.

Queries: asking for clauses that have no label

FORMS works when the document says “Effective Date:” next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an Alias you choose, and the response pairs QUERY blocks with QUERY_RESULT blocks through an ANSWER relationship. The alias is what makes this usable downstream: you match on governing_law, not on the exact wording of the question, so you can reword questions without breaking your schema.

QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}

The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow Pages so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.

Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.

Documents that fight back

Clauses that span a page break

Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to locate a clause and the reassembled text to read it.

Two-column reports and email threads

Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.

Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.

PDFs that were never scanned

This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; pdftotext from poppler-utils pulls the embedded layer if there is one:

pdftotext -layout contract.pdf - | head -c 2000

Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.

Confidence scores you can actually act on

Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about which characters are on the page, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.

It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.

When Custom Queries adapters are worth the effort

If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with CreateAdapter, annotate sample documents, train a version with CreateAdapterVersion, then reference it at inference time.

AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}

That Pages string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.

Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.

Where Textract stops and a language model starts

This is a real decision now, so both sides deserve a hearing. Textract’s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model’s reasoning.

The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. “Does this agreement contain an assignment restriction, and if so summarise it” is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody’s training set.

Most production pipelines use both, which is what AWS’s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.

Troubleshooting

• Extraction stops partway through long documents. You are not following NextToken. Compare the highest page number in your blocks against the source page count.
• Fields come back empty on forms you know were filled in. The value is a SELECTION_ELEMENT and you are only reading WORD children.
• The document is rejected as too large. You are on the synchronous API. Multi-page PDFs must go through StartDocumentAnalysis, and the 10 MB cap is on in-memory size, not file size on disk.
• Textract cannot read the document at all. Check for password protection and XFA-based forms. Both are documented as unsupported.
• The service cannot access the S3 object. Usually the execution role missing s3:GetObject, or a bucket in a different region from the Textract endpoint you called.
• Throttling under batch load. Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.
• Garbled text on reports that look fine to you. Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.

Common mistakes

• Testing only on short documents, which hides the pagination bug entirely.
• Enabling every feature type on every page because it is easier than deciding.
• Running OCR over born-digital PDFs that already carry a text layer.
• Treating a confidence score as a correctness score.
• Scanning the block array linearly instead of building an ID map.
• Polling for job completion inside a Lambda function instead of using SNS.
• Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.

Best practices

• Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.
• Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.
• Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.
• Use aliases on every query and treat them as your schema contract.
• Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.
• Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.
• Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.

Frequently asked questions

Can Amazon Textract handle multi-page PDF contracts?

Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.

Why is my Textract output missing the end of the document?

Almost certainly because you read the first response from GetDocumentAnalysis and stopped. Results are paginated with a NextToken and the job status still reports success, so nothing signals the truncation. Loop until NextToken is absent.

How many Textract Queries can I ask per page?

AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query’s page range is worth the effort.

Does Textract work on handwriting and signatures?

It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.

What languages does Amazon Textract support?

Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.

Should I use Textract or a generative model?

Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.

The one thing worth remembering

Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.

So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.

Need a document extraction pipeline that does not lose fields?

I build and fix document processing pipelines on AWS. Typical work looks like this:

• Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries
• Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries
• Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range
• Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions
• Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension
• Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring

If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.

The post Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports appeared first on John Nessime.

Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify’s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.

That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.

That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn’t there.

This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.

What the pipe actually looks like

Four moving parts, and only two of them live in your account.

1. A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.
2. Shopify creates a partner event source inside your AWS account, in the region you nominated.
3. You associate that source with a partner event bus. This is the step everyone forgets.
4. Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.

The ARN trips people up more than anything else in the setup. Shopify wants the event source ARN, not the event bus ARN. They look similar and only one of them works:

# Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/<id>/<source-name>

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/<id>/<source-name>

Associating the source is a single call, and both the name and the source name are the same string:

# Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --event-source-name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/<id>/<source-name>" 
  --region eu-west-1

The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge’s semantics, not the store’s.

Failure family one: events that never existed

This is the one from the opening, and it is the most expensive because it is completely silent on both sides.

Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.

The same class of hole opens up in two other ways:

• Region mismatch. The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.
• Environment drift. A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.

The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls describe-event-source and alarms if State is anything other than ACTIVE costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.

The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on MatchedEvents hitting zero, not just on errors.

Failure family two: events that arrive twice, or backwards

EventBridge is at-least-once. Shopify’s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.

The duplicate is the obvious one. The same orders/create arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts X-Shopify-Webhook-Id into detail.metadata, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.

The out-of-order case is the one that costs you money quietly. An orders/updated carrying a cancelled status arrives before the orders/updated carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.

The envelope carries what you need for this too. detail.metadata includes X-Shopify-Triggered-At, and the resource in detail.payload carries its own updated_at. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.

The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.

One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.

Failure family three: rules that match nothing

Every Shopify event, regardless of topic, arrives with the same detail-type. That single fact invalidates the routing instinct most people bring from AWS service events.

{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/<id>/<source-name>",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}

Two things to take from that envelope. The resource body is nested under detail.payload, not at the top of detail, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in detail.metadata, which is where all your routing has to happen.

// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}

Do not deploy a pattern you have not tested against a real envelope. test-event-pattern answers in a second and saves an afternoon:

aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json

One rule or thirty?

There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.

What you give up is per-topic visibility. MatchedEvents, FailedInvocations and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.

The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.

Failure family four: events that arrive and die at the target

By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.

First, a whole class of errors gets no retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.

Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.

aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/<id>/<source-name>" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'

Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.

The DLQ permission trap

This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through PutTargets — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.

{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}

The metric that catches this is InvocationsFailedToBeSentToDlq. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.

Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.

Failure family five: the bill

EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.

This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.

Three levers, roughly in order of how much they return:

• Trim at the subscription. Shopify’s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.
• Subscribe to fewer topics. Broad topics like orders/updated fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.
• Archive selectively, and set retention. Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.

One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.

The layer nobody builds until they need it

Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that reached the bus. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.

For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.

• Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.
• Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.
• Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.
• Emit the drift as a metric, not just a log line. “Orders in Shopify but not in our store, last hour” is a graph worth putting on a dashboard, and it should normally read zero.

The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.

Troubleshooting by symptom

Work these in order. Each one is cheap and rules out a whole branch.

Nothing is arriving at all

1. Run describe-event-source. If State is not ACTIVE, stop here. Everything published so far is gone and you need the reconciliation path.
2. Confirm the region of the bus matches the region in the source ARN.
3. List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.
4. Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.
5. Check MatchedEvents on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.

Some topics arrive, order or customer topics do not

This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.

The rule matches but the target does nothing

Compare MatchedEvents against SuccessfulInvocationAttempts on the rule. A gap sends you to FailedInvocations and to the DLQ. Check the target’s resource policy, and check InvocationsFailedToBeSentToDlq before you trust that the DLQ is catching anything.

Events arrive, but late

Look at ThrottledRules and at IngestionToInvocationSuccessLatency. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.

Common mistakes

• Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.
• Registering the event bus ARN instead of the event source ARN, then debugging Shopify’s rejection for an hour.
• Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under detail.payload.
• Routing on detail-type. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.
• Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.
• Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.
• Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.
• Skipping reconciliation because the pipeline “works”. It works right up until it doesn’t, and that is precisely when you need the other path.

Best practices for streaming Shopify events into AWS

• Alarm on the event source state and on MatchedEvents reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.
• Dedupe on X-Shopify-Webhook-Id and version-check on X-Shopify-Triggered-At. Two mechanisms, two problems.
• Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.
• Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.
• Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.
• Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.
• Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.
• Build reconciliation before your first peak trading period, not after your first missing-order ticket.

Frequently asked questions

Do I still need to verify the HMAC signature on the EventBridge path?

No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.

Should I use EventBridge or plain HTTPS webhooks?

HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.

Can I use one partner event source for multiple stores?

Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by X-Shopify-Shop-Domain in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.

Why do I get duplicate order events even though nothing failed?

Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several orders/updated events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.

What happens to events published while my consumer is broken?

They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.

Can I archive and replay Shopify events?

Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.

Does this work the same way for BigCommerce or commercetools?

The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.

The one thing worth remembering

Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.

The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.

Need a hand with your event pipeline?

Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:

• Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.
• Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.
• Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.
• Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.
• Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.
• Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.

If you have a rule pattern that isn’t matching, a DLQ that’s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I’ll tell you what I’d look at first.

The post Streaming Shopify Events into AWS Without Losing Orders appeared first on John Nessime.