Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.

That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.

Why external documents break pipelines that passed testing

Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:

• Container. One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.
• Text origin. Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.
• Layout stability. The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.

You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.

Failure family one: the file is out of spec before Textract sees it

Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.

Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.

Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. BadDocumentException means Textract could not read the file, and retrying fails identically. DocumentTooLargeException means route to the asynchronous path or split the file. InvalidS3ObjectException is usually IAM or a bucket-region mismatch, not a document problem. Only InternalServerErrorException deserves a retry with backoff.

Failure family two: synchronous versus asynchronous is a correctness gate

This is the one that silently eats data, so it goes early.

The synchronous operations, DetectDocumentText and AnalyzeDocument, cap at 10 MB in memory. For PDF and TIFF input they also cap at one page. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.

The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.

A synchronous call looks like this. The document lives in S3, and you name the features you want:

aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1

The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:

import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]

Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:

blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break

Check JobStatus before you trust Blocks. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.

Failure family three: the response is a graph, not a record

Textract does not return fields. It returns a flat list of Block objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.

For form data, the shape is specific. A key-value pair is two blocks of type KEY_VALUE_SET, distinguished by their EntityTypes array containing either KEY or VALUE. The key block carries a relationship of type VALUE holding the ID of its value block, and a relationship of type CHILD holding the IDs of the WORD blocks that spell out the label. The value block carries only CHILD relationships pointing at its own words. So reading one field means three hops.

blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])

Two things about that snippet. The SELECTION_ELEMENT branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.

Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.

Before hand-rolling any of this, look at the AWS-maintained parsers. The amazon-textract-response-parser package and the higher-level amazon-textract-textractor library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.

Getting Amazon Textract structured data out of the right API

Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting TABLES on a corpus with no tables costs money and returns noise.

• DetectDocumentText. Raw text and geometry. Use it when you only need a search index or a text layer.
• AnalyzeDocument with FORMS. Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.
• AnalyzeDocument with TABLES. Grid reconstruction with row and column indices. The right call for line-item grids and statements.
• AnalyzeDocument with QUERIES. You ask natural language questions and get answers back. This is the fix for layout drift.
• AnalyzeDocument with LAYOUT and SIGNATURES. Reading order and structural elements; signature locations for compliance checks.
• AnalyzeExpense. Purpose-built for invoices and receipts, with normalized field types.
• AnalyzeID. Purpose-built for identity documents.

Why Queries beats FORMS on documents you don’t control

FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor’s wording. When they change “P.O. No.” to “Order Reference”, your parser looks for a key that no longer exists and writes a null.

Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:

aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1

The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.

Invoices are their own problem

For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each ExpenseDocument holds SummaryFields for header-level data and LineItemGroups containing the line items. Fields carry a normalized Type such as VENDOR_NAME, plus a ValueDetection and an optional LabelDetection.

That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.

Failure family four: confidence is per field and it is not your risk model

Every extracted element carries a confidence score. Two mistakes follow from that.

The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.

The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:

1. Auto-accept. Above threshold and passing a format check. Straight into the target table.
2. Review. Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.
3. Reject. Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.

Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.

Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.

Failure family five: reprocessing you didn’t budget for

Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline’s life, because that is what parsers do.

So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.

import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"

Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.

One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&O SafeErase exist for exactly that gap between “moved to trash” and “actually gone”.

Where Textract stops being the right tool

Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.

Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.

The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.

Troubleshooting

• Only the first page came through. A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.
• Output truncates partway through a long document. Missing continuation token handling on the get operation. Loop until the token is absent.
• Fields visible on the page never appear in output. The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.
• Quality collapsed for one sender only. Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.
• Throttling during a backfill. Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.
• Table cells land in the wrong columns. Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.

Common mistakes

• Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.
• Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.
• Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.
• Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.
• Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.

Best practices

• Classify before you extract. Route each document to the narrowest API that covers it.
• Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.
• Store raw responses immutably, keyed by content hash, and parse from storage.
• Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.
• Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.
• Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.
• Keep a golden set of real documents with known correct answers, and run it on every parser change.

Frequently asked questions

Can Amazon Textract process multi-page PDFs?

Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.

What is the difference between FORMS and Queries?

FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.

How much does Amazon Textract cost?

Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.

Does Textract handle handwriting?

It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.

Which languages does Amazon Textract support?

Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.

Should I use Textract or Bedrock Data Automation?

Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.

Conclusion

If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.

So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.

Need help turning documents into a pipeline you can trust?

Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:

• Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.
• Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.
• Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.
• Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.
• Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.
• Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.

If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.

The post Turning External Documents Into Structured Data With Amazon Textract appeared first on John Nessime.

That’s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn’t the one that throws. It’s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.

This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.

Why a backfill that finishes cleanly can still be wrong

A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it’s over. The only evidence it leaves behind is a set of objects in a bucket.

So the question you need to answer months later isn’t “did the job succeed?” It’s “is this window complete, and how do I know?” If your design can’t answer that from artifacts on disk, you don’t have a backfill. You have a large amount of data of unknown provenance.

Failure family 1: the source moved while you paginated

This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.

If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.

What to do instead

• Prefer keyset or cursor pagination. If the API offers an opaque cursor or a “records after ID X” parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can’t shift it.
• Sort by an immutable column. If you’re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.
• Bound every request on both ends. Don’t ask for “everything before now.” Ask for a closed interval with an upper bound already in the past. A closed window can’t grow while you read it.
• Know which timestamp the filter applies to. Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.

That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it’s worth writing down which one you chose.

Failure family 2: the retry that wrote the same page twice

Backfills retry. They have to, because you’re hammering an API for hours and something will time out. The problem is what the retry writes.

If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.

import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise

Two things in there are worth explaining rather than copying.

IfNoneMatch="*" is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 PreconditionFailed otherwise. It’s supported on PutObject and CompleteMultipartUpload. That turns “did I already write this page?” into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 ConditionalRequestConflict, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.

The same thing from the CLI, useful for testing the behaviour by hand:

aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"

The retries block matters for a reason that trips people up. In botocore, max_attempts means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a Config object it counts retries on top of the first. total_max_attempts exists only in a Config object and always means total, so using it removes the ambiguity. There’s also an adaptive mode that layers client-side rate limiting on top of standard; AWS still labels it experimental, so I reach for standard plus my own concurrency cap first.

Failure family 3: a layout designed for writing, not reading

The prefix structure you pick during the backfill is the one you’re stuck with. Reorganising later means rewriting every object, and copy requests aren’t free.

Partition on event time, not ingestion time

The tempting layout is the date you ran the job. It’s easy, it’s monotonic, and it’s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:

s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-<token>.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-<n>.parquet

Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.

Don’t let the write path throttle itself

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is partitioned. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 SlowDown responses even when your sustained rate looks well inside the documented ceiling.

For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise within a single day and drive thousands of writes per second into one dt= prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.

Register partitions without a scan

A backfill can create thousands of partitions at once. MSCK REPAIR TABLE handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.

Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.

TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)

Replace START_DATE with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching storage.location.template, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.

Failure family 4: the seam between backfill and live pipeline

Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.

The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don’t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API’s own filtering will all conspire to open a hole at exactly T.

Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.

Failure family 5: the storage class you can’t undo

Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.

• S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.
• Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you’re charged the remainder.
• Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.

Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you’re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.

What I’d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.

One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It’s a five-minute change people usually discover after the bill.

Proving the backfill is complete

Here’s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.

{
  "dataset": "orders",
  "window": "<dt value>",
  "filter_field": "created_at",
  "range_start_inclusive": "<iso8601>",
  "range_end_exclusive": "<iso8601>",
  "pages_written": "<int>",
  "records_written": "<int>",
  "source_reported_total": "<int or null>",
  "last_cursor": "<opaque token>",
  "schema_fingerprint": "<sha256 of sorted field names>"
}

The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose records_written disagrees with source_reported_total is suspect. A window whose schema_fingerprint differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.

Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.

Troubleshooting a backfill that already ran

1. Count objects per window and look at the shape. Use S3 Inventory rather than a live ListObjectsV2 walk on a large bucket. Plot objects per dt= partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.
2. Reconcile record counts against the source. Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what’s in S3. If you kept manifests, this is one query.
3. Check for duplicate natural keys within a partition. A GROUP BY on the record ID with HAVING COUNT(*) > 1 finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren’t deterministic.
4. Look at boundary hours, not whole days. Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.
5. Verify the layout still matches the table definition. If you’re using partition projection and a query returns zero rows, the first suspect is storage.location.template not matching the actual prefix, trailing slashes included.

Common mistakes

• Treating an exit code of zero as evidence of completeness.
• Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.
• Partitioning by ingestion date because it’s what the job naturally knows.
• Making the backfill and live-ingest boundary exact instead of overlapping.
• Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.
• Running MSCK REPAIR TABLE over thousands of new partitions and trusting that it completed.
• Backfilling on a last-modified field while assuming you captured creation events.
• Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.

Best practices when you backfill historical API data into S3

• Make every window independently re-runnable. If re-running one day is routine rather than scary, you’ll actually fix problems when you find them.
• Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.
• Bound every API request on both ends, and never let the upper bound be “now”.
• Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn’t possible.
• Emit a manifest per window, written last, and reconcile it against source counts.
• Compact to Parquet before applying any lifecycle transition.
• Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won’t sleep halfway through.
• Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.

FAQ

Should I write raw JSON or convert to Parquet during the backfill?

Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don’t want to re-fetch years of data because of a type error.

How large should each object be?

In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.

Can I run the backfill in parallel across date windows?

Yes, and that’s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API’s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or Retry-After header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.

Do I need AWS Glue or Airflow for this?

Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.

What if the API caps how far back I can query?

Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it’s usually faster and less rate-limited than paginating.

Does any of this change with S3-compatible storage instead of AWS?

The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with If-None-Match, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you’re depending on against that provider’s documentation rather than assuming parity with S3.

How do I know the backfill is actually finished?

When every expected window has a manifest, every manifest’s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.

The one thing worth remembering

A backfill isn’t finished when the job exits. It’s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.

If you’re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.

Need a hand with a backfill?

This is a large part of what I do. If you’re staring at a historical range and an API with a rate limit, I can help with:

• Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.
• Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.
• Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.
• Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.
• Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.
• Wiring the seam between the backfill and your live pipeline so there’s no gap at the handover, plus the monitoring that would have caught it.

Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That’s usually enough to spot where the gap is.

The post Backfilling Historical API Data into S3 Without Silent Gaps appeared first on John Nessime.

The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.

That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.

Why Amazon Textract data extraction is not just OCR

Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of Block objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.

That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the KEY_VALUE_SET block with entity type KEY, follow its VALUE relationship to another block, then follow that block’s CHILD relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.

Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.

Synchronous or asynchronous: the choice is made for you

People reach for AnalyzeDocument first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at one page. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.

One page. That single constraint decides your architecture. Any real contract goes through StartDocumentAnalysis and GetDocumentAnalysis, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.

Three other constraints from the same quota page are worth knowing before you promise anything to a client:

• Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.
• Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.
• Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.

Pass a ClientRequestToken on submission. It is an idempotency token: the same token returns the same JobId instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.

import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)

The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda’s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.

The result set you never finished reading

This is the one that cost the termination clauses. GetDocumentAnalysis returns results in pages. When there are more blocks than fit in one response, the response carries a NextToken and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. JobStatus still reports success, because the job did succeed. Your code just stopped reading.

Short test documents fit in a single response. That is exactly why this survives testing and dies in production.

def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks

Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.

Choosing feature types without paying for all of them

AnalyzeDocument and StartDocumentAnalysis take a FeatureTypes list: FORMS, TABLES, QUERIES, SIGNATURES and LAYOUT. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.

• FORMS finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.
• TABLES reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.
• QUERIES lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.
• SIGNATURES returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.
• LAYOUT groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.

A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.

Reading the block graph without losing fields

Here is the failure that produces silent empty strings. A key-value pair’s value is not always text. If the value is a tick box, the value block’s children include a SELECTION_ELEMENT block, and that block has no text at all. It has a SelectionStatus of SELECTED or NOT_SELECTED. Concatenate only WORD children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.

def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

Build block_map once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.

Queries: asking for clauses that have no label

FORMS works when the document says “Effective Date:” next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an Alias you choose, and the response pairs QUERY blocks with QUERY_RESULT blocks through an ANSWER relationship. The alias is what makes this usable downstream: you match on governing_law, not on the exact wording of the question, so you can reword questions without breaking your schema.

QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}

The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow Pages so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.

Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.

Documents that fight back

Clauses that span a page break

Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to locate a clause and the reassembled text to read it.

Two-column reports and email threads

Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.

Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.

PDFs that were never scanned

This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; pdftotext from poppler-utils pulls the embedded layer if there is one:

pdftotext -layout contract.pdf - | head -c 2000

Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.

Confidence scores you can actually act on

Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about which characters are on the page, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.

It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.

When Custom Queries adapters are worth the effort

If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with CreateAdapter, annotate sample documents, train a version with CreateAdapterVersion, then reference it at inference time.

AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}

That Pages string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.

Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.

Where Textract stops and a language model starts

This is a real decision now, so both sides deserve a hearing. Textract’s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model’s reasoning.

The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. “Does this agreement contain an assignment restriction, and if so summarise it” is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody’s training set.

Most production pipelines use both, which is what AWS’s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.

Troubleshooting

• Extraction stops partway through long documents. You are not following NextToken. Compare the highest page number in your blocks against the source page count.
• Fields come back empty on forms you know were filled in. The value is a SELECTION_ELEMENT and you are only reading WORD children.
• The document is rejected as too large. You are on the synchronous API. Multi-page PDFs must go through StartDocumentAnalysis, and the 10 MB cap is on in-memory size, not file size on disk.
• Textract cannot read the document at all. Check for password protection and XFA-based forms. Both are documented as unsupported.
• The service cannot access the S3 object. Usually the execution role missing s3:GetObject, or a bucket in a different region from the Textract endpoint you called.
• Throttling under batch load. Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.
• Garbled text on reports that look fine to you. Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.

Common mistakes

• Testing only on short documents, which hides the pagination bug entirely.
• Enabling every feature type on every page because it is easier than deciding.
• Running OCR over born-digital PDFs that already carry a text layer.
• Treating a confidence score as a correctness score.
• Scanning the block array linearly instead of building an ID map.
• Polling for job completion inside a Lambda function instead of using SNS.
• Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.

Best practices

• Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.
• Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.
• Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.
• Use aliases on every query and treat them as your schema contract.
• Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.
• Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.
• Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.

Frequently asked questions

Can Amazon Textract handle multi-page PDF contracts?

Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.

Why is my Textract output missing the end of the document?

Almost certainly because you read the first response from GetDocumentAnalysis and stopped. Results are paginated with a NextToken and the job status still reports success, so nothing signals the truncation. Loop until NextToken is absent.

How many Textract Queries can I ask per page?

AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query’s page range is worth the effort.

Does Textract work on handwriting and signatures?

It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.

What languages does Amazon Textract support?

Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.

Should I use Textract or a generative model?

Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.

The one thing worth remembering

Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.

So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.

Need a document extraction pipeline that does not lose fields?

I build and fix document processing pipelines on AWS. Typical work looks like this:

• Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries
• Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries
• Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range
• Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions
• Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension
• Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring

If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.

The post Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports appeared first on John Nessime.